Data labeling method and device

Abstract

The embodiment of the invention discloses a data annotation method and device. The method comprises the steps that firstly, any target sample in a to-be-labeled first sample set is processed through n first large models, corresponding n labeling results are obtained, and each labeling result comprises a labeled category label and a corresponding labeling basis; and then, processing the n labeling results by using a second large model to obtain a sorting result of m candidate labels involved in the n labeling results. Then, m confidence coefficients corresponding to the m candidate tags are determined, and any ith confidence coefficient is positively correlated with the sorting weight of the ith candidate tag determined based on the sorting result and negatively correlated with the confusion degree when the second big model generates the ith candidate tag; and then, under the condition that the maximum value in the m confidence coefficients is greater than a preset threshold value, determining the candidate label corresponding to the maximum value as the final label of the target sample.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and more particularly to a data annotation method and apparatus, a computer-readable storage medium, and a computing device. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of large-scale models in numerous fields, the demand for high-quality training and optimization data is becoming increasingly urgent. During model training, data quality directly affects the upper limit of the model's learning performance; high-quality labeled data can significantly improve the performance of large-scale models on specific tasks. After model deployment, the labeling and feedback of the model's output becomes a crucial step in its continuous iterative optimization. By constructing a closed-loop optimization mechanism of "output-labeling-feedback," the generated results of large-scale models can be evaluated and corrected, thereby continuously adjusting model parameters and generation strategies to gradually improve output quality. Therefore, whether in the initial training phase or in subsequent continuous optimization, high-quality data labeling is a necessary condition for ensuring that large-scale models fully realize their performance, and its importance becomes increasingly prominent with the development of large-scale model technology.

[0003] However, traditional data annotation schemes face many limitations in practical applications. Therefore, there is an urgent need for a new annotation scheme that can meet the higher requirements of practical applications, such as ensuring both annotation efficiency and annotation quality. Summary of the Invention

[0004] This specification describes a data annotation method and apparatus that can solve the above-mentioned technical problems.

[0005] According to the first aspect, a data annotation method is provided. The method includes: for any target sample in a first sample set to be annotated, processing it using n first-largest models respectively to obtain n corresponding annotation results, where each annotation result includes an annotated category label and a corresponding annotation basis. Processing the n annotation results using a second-largest model to obtain a ranking result of m candidate labels involved in the n annotation results. Determining m confidence levels corresponding to the m candidate labels, where any i-th confidence level is positively correlated with the ranking weight of the i-th candidate label determined based on the ranking result, and negatively correlated with the perplexity of the second-largest model when generating the i-th candidate label. If the maximum value among the m confidence levels is greater than a predetermined threshold, the candidate label corresponding to the maximum value is determined as the final label of the target sample.

[0006] In one embodiment, the sample is processed using n primary models to obtain n corresponding labeled results. This includes inputting a first prompt word into each of the n primary models to obtain the n labeled results. The first prompt word includes the original data of the target sample, a preset classification task description, and multiple preset category labels.

[0007] In one embodiment, determining the m confidence scores corresponding to the m candidate labels includes: determining the m ranking weights corresponding to the m candidate labels, where the higher the position of the i-th candidate label in the ranking result, the higher its i-th ranking weight; determining the m perplexity scores corresponding to the m candidate labels, where the i-th perplexity score is negatively correlated with the generation probability of each word in the i-th candidate label by the second large model; and determining the m confidence scores based on the m ranking weights and m perplexity scores.

[0008] Furthermore, in a specific embodiment, determining the m confidence scores based on the m ranking weights and m perplexities includes: determining the i-th confidence score based on the i-th ranking weight and the i-th perplexity; and normalizing the m confidence scores to obtain the m confidence scores.

[0009] In one embodiment, determining the first sample set includes: classifying each sample in the second sample set using a third model to obtain several pre-classification labels, wherein the several pre-classification labels belong to multiple preset category labels. Based on the several pre-classification labels, each sample is assigned to multiple sample subsets corresponding to the multiple preset category labels. Stratified sampling is performed on the multiple sample subsets to obtain the first sample set.

[0010] Furthermore, in a specific embodiment, for each sample in the second sample set, a third model is used to classify them to obtain several corresponding pre-classification labels, including: for each sample, the third model is used to classify them to obtain several classification probabilities corresponding to the multiple preset category labels; and the preset classification labels corresponding to the classification probabilities that are greater than the probability threshold are classified into several pre-classification labels.

[0011] Furthermore, in one example, stratified sampling is performed on the plurality of sample subsets to obtain the first sample set, including: performing the stratified sampling based on the desired data distribution to obtain the first sample set.

[0012] Furthermore, after determining the candidate label corresponding to the maximum value as the final label of the target sample, the method further includes: constructing corresponding labeled samples using the original data of the target sample and the final label, and including them in the labeled sample set; statistically analyzing the data distribution of the labeled sample set, and if the data distribution does not match the expected data portion, performing incremental sampling based on the multiple sample subsets; and performing labeling processing on the incrementally sampled samples to supplement the labeled sample set.

[0013] According to the second aspect, a data annotation device is provided. The device includes: a multi-model annotation module configured to process any target sample in a first sample set using n first-largest models to obtain n corresponding annotation results, where each annotation result includes an annotated category label and a corresponding annotation basis; a comprehensive analysis module configured to process the n annotation results using a second-largest model to obtain a ranking result of m candidate labels involved in the n annotation results; a confidence determination module configured to determine m confidence levels corresponding to the m candidate labels, where any i-th confidence level is positively correlated with the ranking weight of the i-th candidate label determined based on the ranking result, and negatively correlated with the perplexity when the second-largest model generates the i-th candidate label; and a confidence splitting module configured to determine the candidate label corresponding to the maximum value among the m confidence levels as the final label of the target sample if the maximum value among the m confidence levels is greater than a predetermined threshold.

[0014] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method provided in the first aspect.

[0015] According to a fourth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method provided in the first aspect.

[0016] In summary, the data annotation method and apparatus disclosed in the embodiments of this specification can simultaneously improve annotation efficiency and annotation accuracy. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1This is one of the implementation architecture diagrams of the improved data annotation scheme disclosed in the embodiments of this specification;

[0019] Figure 2 This is a schematic diagram of the flow steps of the data annotation method disclosed in the embodiments of this specification;

[0020] Figure 3 This is the second implementation architecture diagram of the improved data annotation scheme disclosed in the embodiments of this specification;

[0021] Figure 4 This is a functional structure diagram of the data annotation device disclosed in the embodiments of this specification. Detailed Implementation

[0022] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0023] As mentioned earlier, traditional data annotation schemes face many limitations in practical applications. Specifically, currently, data annotation mainly relies on two methods: manual annotation and semi-automatic annotation.

[0024] Traditional manual annotation requires a large number of annotators to label each data point individually. This method suffers from high annotation costs, low annotation efficiency, and inconsistent annotation standards, resulting in high model training costs and difficulty in meeting the requirements of rapid iteration.

[0025] To address these issues, the industry has proposed several semi-automatic annotation methods. For example, rule-based annotation methods initially label data using pre-defined rules, followed by manual review and correction. Traditional machine learning-based annotation methods train classifiers using existing labeled data to predict unlabeled data. However, these methods perform poorly in complex scenarios, have limited generalization capabilities, and require extensive domain knowledge for rule design or feature engineering, making them difficult to adapt to rapidly changing application needs. Large-model-based annotation methods utilize prompt word engineering and the zero-shot / few-shot learning capabilities of large models to guide them in generating annotation results, which are then manually checked and corrected. However, current large-model-based data annotation schemes still suffer from the following problems:

[0026] 1) Low annotation quality: Due to the illusion inherent in large models, the generated annotation results may contain factual errors, logical inconsistencies, or fail to meet specific domain standards, thus affecting the annotation quality.

[0027] 2) Lack of a reasonable assessment scheme for the uncertainty of annotation results: It is difficult to reasonably measure the uncertainty of annotation results for large models, which leads to the only way to verify the annotation results through random sampling. When the sampling rate is low, a large number of mislabeled data will still be missed, affecting data quality; when the sampling rate is high, problems such as high cost, low efficiency and inconsistent standards for manual annotation will be encountered.

[0028] Based on the above observations and analysis, the applicant proposes an improvement scheme that can fully utilize the powerful capabilities of large models to improve annotation efficiency while ensuring annotation quality.

[0029] The implementation architecture of the improved solution can be found in [reference needed]. Figure 1 :

[0030] 1. Multi-model collaborative voting: For any sample to be labeled, multiple large models are used to process it separately, resulting in multiple outputs, each of which includes a predicted label and the corresponding thought process.

[0031] 2. Multi-angle collaborative analysis: Utilize a large model to perform multi-angle comprehensive analysis on multiple outputs to obtain the ranking of predicted labels.

[0032] 3. Confidence Calculation (or Multi-Outcome Weighted Evaluation): The weights of the predicted labels are calculated based on the ranking to characterize the labeling consistency between models, and the uncertainty of the labels within the model is calculated by the perplexity of the label ranking output by the large model. Furthermore, the final confidence score is obtained by combining the ranking weights and the perplexity.

[0033] 4. Calculation based on confidence level: Labeled samples with a confidence level higher than the threshold (including the sample's original data and predicted labels) are included in the final labeled dataset, while labeled samples with a confidence level lower than the threshold are discarded or subject to manual review. If manual review is performed, the reviewed samples will also be included in the labeled dataset.

[0034] The above process introduces: 1) A multi-model voting and consistency enhancement mechanism: Multiple large models are used to label the same task, and a voting mechanism is used to improve the stability of the results and correct labeling errors caused by the illusion of a single model. 2) A confidence-based triage strategy: Voting results, ranking weights, and perplexity indicators are integrated to construct a multi-dimensional confidence scoring system, thereby achieving a good balance between automation rate and accuracy. This allows for the simultaneous improvement of labeling efficiency and accuracy.

[0035] Next, the specific implementation steps of the above improvement plan will be introduced. (See also...) Figure 2 The diagram illustrates the steps of the data annotation method. It should be understood that the entity executing this method can be any device, platform, server, or equipment cluster with computing and processing capabilities. Figure 2 The method steps shown include:

[0036] First, in step S210, for any target sample in the first sample set to be labeled, n first-largest models are used to process it respectively to obtain n corresponding labeling results, where each labeling result includes the labeled category label and the corresponding labeling basis.

[0037] It should be noted that the "first" in "first sample set" and "first large model", as well as similar terms such as "second" and "third" elsewhere in the text, are all used to distinguish similar things and do not have any other limiting function such as ranking.

[0038] The first sample set may include images, text, audio, or video. The corresponding classification tasks or tagging systems can be designed by staff based on the needs of the actual business scenario. One possible business scenario involves classifying electronic documents, for example, into personal documents or company documents. Another possible scenario involves classifying insurance-related documents, such as into medical records, invoices, diagnostic certificates, etc. It should be noted that all user-related data mentioned in this document was legally collected and used after obtaining user authorization.

[0039] The aforementioned target sample can refer to any sample in the first sample set.

[0040] The aforementioned "n first-class models" refer to multiple first-class models, meaning that n is greater than 1; for example, n=3. These multiple first-class models differ from each other, for example, in terms of different training data, model structures, or knowledge coverage and inference styles. These multiple first-class models can include open-source or self-developed large models. Therefore, by introducing a mechanism for collaborative annotation of multiple large language models, integrating multiple diverse large models to independently complete the annotation task for each sample, this heterogeneous design allows for multi-dimensional examination of the same data from the perspectives of different models. This avoids annotation errors caused by a single model due to knowledge blind spots, cue sensitivity, or inference bias, thereby improving the consistency and accuracy of the annotation results.

[0041] The input to each primary model includes the raw data (or raw feature data) of the target samples. Furthermore, the raw data can be populated into a pre-defined prompt word module, serving as a unified input for each primary model. The prompt word template can include a classification task description and a labeling system.

[0042] It should be understood that the classification task description can be a standardized task description set by staff to guide the large model to perform a classification task (e.g., selecting a unique answer from a preset category), and to provide relevant reasons and explanations (or labeling criteria). For example, the task description could be "Please determine which category this file belongs to and provide your thought process."

[0043] The aforementioned tagging system can include multiple category tags, such as private documents or company documents; in addition, it can also include descriptions of each category tag, for example, the description of company documents can be: documents related to business operations, contracts, project reports, and internal communications.

[0044] In this way, each primary model receives the same input content, and the unified prompt word template guides each model to follow a consistent output format, ensuring the comparability and structured processing of the annotation results. Unlike traditional automatic annotation methods that only focus on the final label, this step particularly emphasizes the interpretability of the annotation process. Each primary model must include the reasoning behind its judgment along with the output annotation results. This "answer + reason" output mode allows subsequent comprehensive analysis to not only compare whether the labels are consistent, but also to evaluate the rationality of each model's reasoning logic, the completeness of the evidence chain, and whether there are any illusions or far-fetched interpretations.

[0045] Each large model, after processing the input to the target sample, can obtain corresponding annotation results, including labeled categories and corresponding annotation criteria. The labeled categories can be understood as candidate labels or predicted labels, which belong to multiple pre-defined category labels within the labeling system. Furthermore, the annotation criteria can include keywords, contextual logic, or judgment rules, etc.

[0046] From the above, we can obtain n annotation results corresponding to the target sample.

[0047] Next, in step S220, the n annotation results are processed using the second large model to obtain the ranking results of the m candidate labels involved in the n annotation results. It can be understood that 1≤m≤n.

[0048] It should be understood that the second largest model can be different from the n first largest models mentioned above, or it can be one of the n first largest models mentioned above. Assuming that the second largest model is one of the n first largest models mentioned above, in this case, the execution of this step will not use the context generated by the second largest model performing the classification and labeling task in step S210.

[0049] Specifically, in this step, the above n labeled results and the task description for comprehensive analysis can be input into the second large model to obtain the ranking result. The task description guides the second large model to systematically integrate the n independent labeled results, including voting on candidate labels and, based on the labeling criteria, evaluating the rationality of each model's reasoning logic, the completeness of the evidence chain, and whether there are any illusions or far-fetched interpretations. Thus, through comprehensive analysis, a priority-based label ranking is obtained, with higher-ranked labels having higher priority, thereby providing a basis for subsequent confidence calculations and intelligent triage.

[0050] For example, assuming that n annotation results are 3 annotation results, and the corresponding labels are A, A, and B respectively, after this step, the second model may output a label sorting as A→B, which means that it is more reasonable to label the target sample as label A.

[0051] It should be noted that the output of the second model can also include explanations of the sorting results, which can be used to assist in any subsequent manual review.

[0052] From the above, we can obtain the sorting results of the m candidate labels involved in the n annotation results of the target sample.

[0053] Then, in step S230, m confidence levels corresponding to the m candidate labels are determined, wherein any i-th confidence level is positively correlated with the ranking weight of the i-th candidate label determined based on the ranking result, and negatively correlated with the perplexity when the second large model generates the i-th candidate label.

[0054] It should be noted that the confidence measurement in this step includes calculating the ranking weight and model perplexity separately, thereby balancing the consistency of annotations between models and the determinism of annotations within models.

[0055] Specifically, this step may include the following sub-steps S231-S233 ( Figure 2 (not shown in the image)

[0056] Step S231: Determine the m ranking weights corresponding to the m candidate labels, where the higher the position of the i-th candidate label in the ranking result, the higher its corresponding i-th ranking weight.

[0057] In one embodiment, for any candidate label, the reciprocal of its ranking in the sorting result can be used to determine the corresponding ranking weight. For example, in the sorting result A→B, the ranking weight of label A is 1, and the ranking weight of label B is 0.5.

[0058] In another embodiment, the ranking weight of candidate tags can be calculated using the following formula:

[0059] (1)

[0060] Where λ is the length of the sorted result, and i is the right-to-left order of the candidate labels in the sorted result. For example, when the output sort is A→B→C, the length of the sort is 3. For label A, its right-to-left order in the sort is 3, so the sort weight of label A is:

[0061] (2)

[0062] Similarly, the ranking weights for label B and label C are as follows:

[0063] (3)

[0064] Based on the above, the ranking weight of each candidate label can be determined according to the ranking results, with the first label having the highest weight and subsequent labels having decreasing weights. Rank weights can measure the consistency of annotations across different large models.

[0065] Step S232: Determine the m perplexity values ​​corresponding to the m candidate labels, wherein the i-th perplexity value is negatively correlated with the generation probability of each word in the i-th candidate label by the second large model.

[0066] It should be noted that perplexity (PPL) measures the uncertainty of the model when generating candidate labels. A higher PPL indicates a more perplexing model and higher uncertainty, while a lower PPL indicates a more certain model in generating labels.

[0067] In one embodiment, the probability distribution entropy of the second-largest model when generating candidate labels can be calculated as the perplexity:

[0068] (4)

[0069] Where X=(x1,x2,...,x N p(x) represents the token sequence corresponding to any candidate tag. For example, assuming the candidate tag is "private document", the token sequence can be (x1, x2) = ("private", "document"), and the length of the token sequence is N = 2. i ) is the second largest model in a given context (i.e., x1 to x) i-1 In the case of ), the i-th word unit is predicted to be x. i The probability of.

[0070] It needs to be understood that p(x) i The second largest model's state data during the output sorting process can be directly collected and used.

[0071] In another embodiment, the perplexity of the candidate label can be calculated using the following formula:

[0072] (5)

[0073] It should be understood that the introduction of mathematical symbols in equation (5) can be found in the relevant introduction of equation (4), and will not be repeated here.

[0074] From the above, we can obtain the perplexity of the second largest model for each candidate label among the m candidate labels.

[0075] Step S233: Determine the m confidence levels based on the m ranking weights and m perplexity levels.

[0076] Specifically, for any i-th candidate label among m candidate labels, the i-th confidence score is determined based on its corresponding i-th ranking weight and i-th perplexity. It should be noted that the i-th confidence score is positively correlated with the i-th ranking weight and negatively correlated with the i-th perplexity. In other words, the earlier the i-th candidate label appears in the ranking, the higher the certainty of the model in generating it, and thus the higher its confidence score.

[0077] For example, the confidence score can be calculated using the following formula:

[0078] (6)

[0079] In the above formula (6), This represents the ranking weight of the i-th candidate label. This represents the perplexity of the i-th candidate label.

[0080] It should be understood that the confidence score can also be calculated using a variation of the above formula (6), for example, by taking the square root of the expression on the right side of the equals sign.

[0081] Furthermore, in one embodiment, the i-th confidence score can be directly used as the i-th confidence level.

[0082] In another embodiment, the m confidence scores corresponding to the m candidate labels can be normalized to obtain the m confidence levels. It should be understood that the normalization process can be implemented using existing normalization functions, such as directly calculating the proportion, or using the Softmax function. The resulting confidence levels will range from [0,1], and the sum of the m confidence levels will be 1.

[0083] From the above, we can obtain m confidence scores corresponding to m candidate labels. These scores comprehensively consider the consistency of annotations among multiple first models and the uncertainty of comprehensive analysis within the second largest model, thus accurately reflecting the probability of the correctness of the candidate labels.

[0084] The above describes how to obtain m confidence scores corresponding to m candidate labels by executing sub-steps S231-S233.

[0085] Then, in step S240, if the maximum value (or maximum confidence level) among the m confidence levels is greater than a predetermined threshold, the candidate label corresponding to the maximum value is determined as the final label of the target sample. It should be understood that the predetermined threshold is a pre-configured confidence threshold, and its specific value can be flexibly adjusted according to actual needs; for example, it can be set to 0.6.

[0086] In this way, arbitrary target samples can be split. Similarly, all samples in the first sample set are divided into two flows: samples with a maximum confidence score higher than the confidence threshold enter the "direct adoption" channel, and the labeled samples constructed from them and their corresponding labels are included in the final labeled dataset (which can be used to train large models), achieving fully automated data labeling. Samples below the confidence threshold can be discarded directly, or they can enter the "manual review" queue for secondary confirmation and correction by professional labelers.

[0087] In addition, the triage mechanism can also adopt a multi-level classification strategy: for example, a maximum confidence level > 0.8 is "high confidence" and does not require verification; 0.6–0.8 is "medium confidence" and can be sampled and inspected; < 0.6 is "low confidence" and requires mandatory manual intervention.

[0088] It should be understood that for data annotation tasks in different scenarios, a portion of the data can be sampled for intelligent annotation, and the intelligently annotated data can be manually reviewed after random sampling. Based on the results of the manual review, a confidence threshold for that scenario can be set.

[0089] Therefore, by introducing multi-model voting and consistency enhancement mechanisms, as well as a confidence-based diversion strategy, into the improved scheme, both annotation efficiency and annotation accuracy can be improved simultaneously.

[0090] According to another embodiment, in the improved solution, it is proposed that... Figure 1 Based on the implementation architecture shown, add Figure 3 The diagram illustrates the data pre-labeling and data sampling stages to ensure that the data distribution in the labeled dataset conforms to expectations. It should be noted that the dataset on which the data pre-labeling is based can be referred to as the second sample set, and the output of the data sampling stage is the first sample set mentioned above. Clearly, the first sample set is a subset of the second sample set.

[0091] 1. Data pre-labeling: The data is pre-labeled using a third model, and all possible classifications are output during the labeling process.

[0092] It should be noted that the core objective of this step is not to pursue high accuracy, but to ensure the completeness and coverage of the labeled categories. Specifically, the improved scheme proposes to further introduce a lightweight classification model (corresponding to the third model mentioned above) as a "breadth detector," which outputs multiple candidate types that each sample in the second sample set may belong to. In this way, a complete semantic candidate space can be provided for subsequent sampling and large-scale model annotation, avoiding the systematic neglect of certain potential categories due to prior bias, cue word limitations, or attention bias, thereby ensuring the comprehensiveness and robustness of the entire annotation process.

[0093] The implementation steps in this stage may include: classifying each sample in the second sample set using a third model to obtain several pre-classification labels, or several pre-labeled labels. It should be understood that "several" in this text refers to one or more in quantity.

[0094] The third model described above can implement a lightweight classification model, such as a lightweight convolutional neural network, a small model obtained through knowledge distillation, and so on. It should be understood that the third model is generally not a large model, but it can also be a large model with relatively few parameters or relatively low structural complexity.

[0095] The labeling system used in the third model is the same as that used in the labeling stage of the large model mentioned above. This means that the pre-labeled labels obtained above belong to multiple preset category labels in the labeling system.

[0096] Furthermore, the above-mentioned classification using the third model to obtain several corresponding pre-classification labels may include: for any sample in the second sample set, processing the sample's original data using the third model to obtain multiple classification probabilities corresponding to multiple preset category labels, and then assigning the preset classification labels corresponding to the classification probabilities greater than a probability threshold to the aforementioned several pre-classification labels. It should be noted that the probability threshold is lower than the threshold in the precise classification scenario (e.g., 0.3), thereby achieving the output of all possible categories.

[0097] In the data pre-labeling stage, the samples in the second sample set can be pre-labeled, all possible classifications can be output, and a preliminary judgment on the category to which the sample belongs can be made.

[0098] 2. Data sampling: Perform stratified sampling according to the required data distribution.

[0099] Specifically, based on the pre-classification labels corresponding to each sample in the second sample set, each sample can be categorized into multiple sample subsets corresponding to multiple preset category labels in the label system, thus achieving stratification of the second sample set. It should be understood that since a sample can correspond to one or more pre-classification labels, it may be categorized into different sample subsets simultaneously.

[0100] Furthermore, based on the desired data distribution, multiple sample subsets can be sampled separately, or in other words, stratified random sampling can be used, with the sampled samples forming the first sample subset. This ensures that samples from all categories are effectively covered, especially low-frequency but critical long-tail categories.

[0101] For example, different sampling weights (or sampling ratios) can be set for multiple sample subsets according to the required proportion of samples of different categories, so as to obtain a first sample set that conforms to the desired distribution.

[0102] The above is an introduction. Figure 3 The data pre-labeling and data sampling steps shown in the figure can make the data distribution in the first sample set meet expectations, thereby making the data distribution of the final labeled dataset close to expectations.

[0103] According to yet another embodiment, in adding Figure 3 Based on the data pre-labeling and data sampling steps shown, a data distribution statistics step can be added after confidence level splitting.

[0104] Data distribution statistics can be the final step in closed-loop management, used to verify whether the labeled data meets the preset training requirements and business distribution requirements. After completing one round of labeling and distribution, the system automatically performs multi-dimensional statistical analysis on the final labeled dataset, including: the distribution of sample quantity in each category, the distribution over time periods, and the distribution of difficulty levels.

[0105] It should be noted that the statistical analysis of the difficulty level distribution may include: statistical analysis of the results of multiple inferences on any labeled sample based on the fourth model. The feature data of each labeled sample will be processed multiple times by the fourth model under the same conditions. By comparing the consistency between the model output and the standard answer (the final label), the frequency of its correct output will be counted. The difficulty level will be divided according to the frequency of correct output: 1) Easy samples: Samples with a high frequency of correct output in multiple inferences by the model, indicating that the model's recognition or processing of the sample is stable and accurate; 2) Medium difficulty samples: Samples with a medium frequency of correct output by the model; 3) Hard samples: Samples with a low frequency of correct output by the model, indicating that the model's prediction of the sample is unstable or prone to error. Then, based on the distribution of the correct output frequency of all labeled samples, the number and proportion of samples at each difficulty level will be automatically classified and counted to obtain the difficulty level distribution of the labeled dataset.

[0106] By comparing the actual distribution with the target distribution (such as business proportion or model training requirements), the system determines whether there is a significant deviation. If there are insufficient samples for a certain category, the system triggers an incremental sampling mechanism: returning to the original data pool (corresponding to the multiple sub-sample sets mentioned above), re-executing the sampling strategy for the missing category, collecting relevant samples, and completing the subsequent annotation process again. This iterative process continues until the labeled dataset meets the required data distribution, at which point the automated annotation process terminates.

[0107] By introducing a data distribution statistics step, we can ensure that the final labeled dataset can truly reflect the overall business situation and avoid the downstream model from exhibiting class bias or reduced generalization ability due to sampling deviation.

[0108] In summary, the data annotation method disclosed in the embodiments of this specification can produce the following beneficial effects:

[0109] 1) Credibility: The improved solution abandons the reliance on a single large model for judgment and instead adopts a multi-model collaborative mechanism. Multiple large models with different knowledge backgrounds independently complete annotation and output reasoning. The large models comprehensively evaluate the consistency of the results and the sufficiency of the reasons, avoiding misjudgments caused by the bias or illusion of a single model. At the same time, perplexity level (PPL) is introduced as a quantitative indicator of the model's internal uncertainty, forming a two-dimensional confidence assessment system combined with external consistency. This system not only focuses on "whether multiple models say the same thing" but also on "whether the model itself is firm in its statement." This combined internal and external assessment method makes the confidence level more discriminative, effectively identifying results that appear reasonable but are actually unreliable.

[0110] 2) Automation: Addressing the issues of high costs and difficulty in handling large-scale data through manual annotation, this solution constructs an intelligent annotation workflow centered on automation. Multiple large language models are used to annotate sampled data in parallel, replacing traditional manual reading and interpretation, significantly improving processing efficiency. More importantly, the system does not rely entirely on manual review; instead, it intelligently distributes data based on confidence levels, ensuring fully automated annotation of high-confidence data while only requiring manual processing for low-confidence results.

[0111] Furthermore, a small-model pre-labeling mechanism can be introduced to make a preliminary judgment on the category to which the sample belongs. Based on this, the original data can be diversified according to the sampling distribution of the target, thereby utilizing multiple large language models to label the sampled data in parallel. In addition, a statistical step of data distribution can be introduced after confidence level splitting to achieve closed-loop feedback.

[0112] Corresponding to the data annotation method described above, this specification also discloses a data annotation device that can be integrated into any server, platform, or device cluster with computing and processing capabilities. Figure 4 The data annotation device 400 shown includes the following functional modules:

[0113] The multi-model annotation module 410 is configured to process any target sample in the first sample set to be annotated using n first-largest models to obtain n corresponding annotation results, where each annotation result includes an annotated category label and the corresponding annotation basis. The comprehensive analysis module 420 is configured to process the n annotation results using a second-largest model to obtain a ranking result of m candidate labels involved in the n annotation results. The confidence determination module 430 is configured to determine m confidence levels corresponding to the m candidate labels, where any i-th confidence level is positively correlated with the ranking weight of the i-th candidate label determined based on the ranking result, and negatively correlated with the perplexity when the second-largest model generates the i-th candidate label. The confidence splitting module 440 is configured to determine the candidate label corresponding to the maximum value among the m confidence levels as the final label of the target sample if the maximum value among the m confidence levels is greater than a predetermined threshold.

[0114] In one embodiment, the multi-model annotation module 410 is specifically configured to: input the first prompt word into the n first large models respectively to obtain the n annotation results; wherein, the first prompt word includes the original data of the target sample and a preset classification task description, as well as multiple preset category labels.

[0115] In one embodiment, the confidence determination module 430 is specifically configured to: determine m ranking weights corresponding to the m candidate labels, wherein the higher the position of the i-th candidate label in the ranking result, the higher its corresponding i-th ranking weight; determine m perplexities corresponding to the m candidate labels, wherein the i-th perplexity is negatively correlated with the generation probability of each word in the i-th candidate label by the second large model; and determine the m confidence levels based on the m ranking weights and m perplexities.

[0116] Furthermore, in a specific embodiment, the confidence determination module 430 is configured to determine the m confidence levels based on the m ranking weights and m perplexities, specifically including: determining the i-th confidence score based on the i-th ranking weight and the i-th perplexity; and normalizing the m confidence scores to obtain the m confidence levels.

[0117] In one embodiment, the data labeling device 400 further includes: a data pre-labeling module 450, configured to classify each sample in the second sample set using a third model to obtain several pre-classification labels, wherein the several pre-classification labels belong to multiple preset category labels; and a data sampling module 460, configured to classify each sample into multiple sample subsets corresponding to the multiple preset category labels based on the several pre-classification labels; and to perform stratified sampling on the multiple sample subsets to obtain the first sample set.

[0118] Furthermore, in a specific embodiment, the data prelabeling module 450 is specifically configured to: classify each sample using the third model to obtain multiple classification probabilities corresponding to the multiple preset category labels; and assign the preset classification labels corresponding to the classification probabilities that are greater than the probability threshold to the multiple pre-classification labels.

[0119] Furthermore, in one example, the data sampling module 460 is specifically configured to perform the hierarchical sampling based on the desired data distribution to obtain the first sample set.

[0120] Furthermore, the data labeling device 400 also includes a data distribution statistics module 470, configured to: construct corresponding labeled samples using the original data of the target sample and the final label, and classify them into a labeled sample set; statistically analyze the data distribution of the labeled sample set, and if the data distribution does not match the expected data portion, perform incremental sampling based on the multiple sample subsets; and perform labeling processing based on the incrementally sampled samples to supplement the labeled sample set.

[0121] It should be noted that for a description of the above-mentioned functional modules or functional units, please refer to the relevant description of the process method in the foregoing embodiments.

[0122] In this specification, the Large Language Model (LLM) may also be referred to simply as the Large Model. A Large Language Model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large Language Models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. Large Language Models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, giving them outstanding generalization capabilities. The core capabilities of Large Language Models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the Large Language Model to generate specific outputs by designing prompts. Cue words can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of large language models. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0123] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0124] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025, public link: https: / / doi.org / 10.48550 / arXiv.2303.18223), and will not be repeated here.

[0125] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform... Figure 2 The method described.

[0126] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement... Figure 2 The method described.

[0127] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0128] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data annotation method, comprising: For any target sample in the first sample set to be labeled, n first-largest models are used to process it respectively to obtain n corresponding labeling results, where each labeling result includes the labeled category label and the corresponding labeling basis; The second large model is used to process the n annotation results to obtain the ranking results of the m candidate labels involved in the n annotation results; Determine m confidence levels corresponding to the m candidate labels, wherein any i-th confidence level is positively correlated with the ranking weight of the i-th candidate label determined based on the ranking result, and negatively correlated with the perplexity when the second large model generates the i-th candidate label; If the maximum value among the m confidence scores is greater than a predetermined threshold, the candidate label corresponding to the maximum value is determined as the final label of the target sample.

2. The method according to claim 1, wherein, By processing n first-largest models respectively, we obtain n corresponding labeled results, including: The first prompt word is input into each of the n first large models to obtain the n annotation results; The first prompt word includes the original data of the target sample, a preset classification task description, and multiple preset category labels.

3. The method according to claim 1, wherein, Determining the m confidence levels corresponding to the m candidate labels includes: Determine the m ranking weights corresponding to the m candidate labels. The earlier the i-th candidate label is in the ranking result, the higher its i-th ranking weight. Determine m perplexity values ​​corresponding to the m candidate labels, where the i-th perplexity value is negatively correlated with the generation probability of each word in the i-th candidate label by the second large model; Based on the m ranking weights and m perplexity levels, the m confidence levels are determined.

4. The method according to claim 3, wherein, Based on the m ranking weights and m perplexity levels, the m confidence levels are determined, including: Based on the i-th ranking weight and the i-th perplexity, determine the i-th confidence score; The m confidence scores are normalized to obtain the m confidence levels.

5. The method according to claim 1, wherein, The determination of the first sample set includes: For each sample in the second sample set, a third model is used to classify them to obtain several pre-classification labels, which belong to multiple preset category labels. Based on the aforementioned pre-classification labels, each sample is assigned to a subset of samples corresponding to the aforementioned preset category labels; The first sample set is obtained by performing stratified sampling on the multiple sample subsets.

6. The method according to claim 5, wherein, For each sample in the second sample set, the third model is used to classify them, resulting in several pre-classification labels, including: For each sample, the third model is used to classify it, and multiple classification probabilities corresponding to the multiple preset category labels are obtained; The preset classification labels corresponding to the classification probabilities that are greater than the probability threshold are classified as the several pre-classification labels.

7. The method according to claim 5, wherein, The first sample set is obtained by performing stratified sampling on the plurality of sample subsets, including: The first sample set is obtained by performing the hierarchical sampling based on the desired data distribution.

8. The method according to claim 7, wherein, After determining the candidate label corresponding to the maximum value as the final label of the target sample, the method further includes: Using the original data of the target sample and the final label, construct the corresponding labeled sample and include it in the labeled sample set; The data distribution of the labeled sample set is statistically analyzed. If the data distribution does not match the expected data portion, incremental sampling is performed based on the multiple sample subsets. The samples obtained from the incremental sampling are labeled to supplement the labeled sample set.

9. A data annotation device, comprising: The multi-model annotation module is configured to process any target sample in the first sample set to be annotated using n first-largest models respectively, and obtain n corresponding annotation results, where each annotation result includes the annotated category label and the corresponding annotation basis; The comprehensive analysis module is configured to process the n annotation results using the second largest model to obtain the ranking results of the m candidate labels involved in the n annotation results; The confidence determination module is configured to determine m confidence levels corresponding to the m candidate labels, wherein any i-th confidence level is positively correlated with the ranking weight of the i-th candidate label determined based on the ranking result, and negatively correlated with the perplexity when the second large model generates the i-th candidate label; The confidence score splitting module is configured to determine the candidate label corresponding to the maximum value among the m confidence scores as the final label of the target sample if the maximum value is greater than a predetermined threshold.

10. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed in the computer, it causes the computer to perform the method of any one of claims 1-8.

11. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-8.

Images