Multi-level human-machine collaborative data labeling method and system based on task ambiguity evaluation

Abstract

The application discloses a kind of multi-level man-machine collaborative data labeling method and system based on task ambiguity evaluation, first constructs and trains ambiguity prediction model, utilizes lightweight ambiguity prediction model, predicts the cognitive ambiguity score and logic ambiguity score of each data instance;Four-stage labeling resource pool containing low-performance large language model, high-performance large language model, ordinary crowd sourcing labeler and expert labeler is constructed;According to the predicted two-dimensional ambiguity score, the data instance is automatically routed to the optimal labeling resource by dynamic scheduling strategy, for the task with high cognitive ambiguity, the upgrading mechanism combining crowd sourcing voting and expert arbitration is used, and the final labeled data set is output.The application can intelligently match labeling cost and ability according to the intrinsic properties of the task, significantly reduce the labeling cost and improve the labeling efficiency under the premise of ensuring the data labeling quality, solve the problem of high cost of traditional crowd sourcing and unstable quality of single model labeling.

Description

Technical Field

[0001] This invention belongs to the technical field of artificial intelligence and data processing, and relates to dynamic task allocation and resource scheduling technology based on task fuzzy quantitative analysis. It mainly relates to a multi-level human-computer collaborative data annotation method and system based on task fuzziness evaluation. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of deep learning algorithms in fields such as natural language processing and computer vision, high-quality labeled data has become a key element in improving model performance. The training effect of a model largely depends on the size and accuracy of the training dataset; therefore, data labeling has become an indispensable and fundamental link in the artificial intelligence industry chain.

[0003] Traditional data annotation primarily relies on a crowdsourcing model. This approach typically involves recruiting a large number of ordinary users or domain experts through internet platforms to label data. While manual annotation is highly accurate when handling tasks requiring complex common sense, sentiment understanding, or domain-specific knowledge, its drawbacks are also significant: First, it is extremely costly, especially in vertical fields requiring expert participation, such as medicine and law, where annotation costs often increase linearly or even exponentially with the amount of data. Second, manual annotation is relatively inefficient, making it difficult to meet the demands of rapid iteration of massive amounts of data in the era of large-scale models. Furthermore, due to the subjective differences among annotators, it is often difficult to reach a consensus on fuzzy tasks, resulting in unstable data quality.

[0004] In recent years, with the emergence of large-scale language models (LLMs) such as GPT-4, automated data annotation using large models—LLM-as-a-Judge or Zero-shot Labeling—has become an emerging technological trend. LLMs possess powerful semantic understanding and generation capabilities, enabling them to process large amounts of data rapidly at extremely low marginal costs. However, relying solely on LLMs for annotation also has significant technical drawbacks: on the one hand, large models inherently suffer from the "illusion" problem, potentially generating seemingly reasonable but actually factually incorrect labels, especially when handling complex logical reasoning or fact-checking tasks, where their accuracy often falls short of human experts; on the other hand, while the API call cost of high-performance large models is lower than manual methods, it remains a considerable expense in large-scale data scenarios, while low-cost small models often lack sufficient performance to meet the requirements of high-quality annotation.

[0005] To combine human accuracy with machine efficiency, some collaborative annotation schemes have emerged in existing technologies. The most mainstream approach currently employs a "static-hybrid" strategy, which involves a fixed workflow: first, the model pre-labels all data, then human annotators perform full verification or correction of the model's pre-labeled results. However, this static strategy suffers from significant resource allocation inconsistencies and waste. For large amounts of simple, clear data samples, the model's pre-labeling accuracy is extremely high, making the full manual verification redundant and wasting human resources. Conversely, for highly subjective or logically complex "fuzzy" samples, the model often provides incorrect judgments or even misleading interpretations. In such cases, human intervention goes beyond simple verification; it requires re-examination and re-labeling, wasting the model's initial computational resources without substantially reducing the human workload.

[0006] Therefore, existing technologies lack a mechanism that can deeply understand the inherent attributes of tasks and dynamically and accurately schedule heterogeneous annotation resources with different capabilities and costs. In other words, how to accurately quantify and evaluate the ambiguity type and level of each data task, and design the optimal task allocation strategy accordingly to maximize annotation quality and efficiency within a limited budget, is a technical problem that urgently needs to be solved in the current data annotation field. Summary of the Invention

[0007] This invention addresses the problems of high costs associated with relying solely on manual annotation, unstable annotation quality when relying solely on large language models, inflexible resource scheduling in existing static human-machine collaboration models, and low annotation efficiency and cost-effectiveness for tasks with different ambiguity types. It proposes a multi-level human-machine collaborative data annotation method and system based on task ambiguity assessment. First, a ambiguity prediction model is constructed and trained, using the feature vectors of sample data as input and two-dimensional ambiguity ground truth labels as output targets for supervised training. The feature vectors of the dataset to be annotated are input into the ambiguity prediction model, which uses a lightweight ambiguity prediction model to predict the cognitive and logical ambiguity scores of each data instance. A four-level annotation resource pool is constructed, including low-performance large language models, high-performance large language models, ordinary crowdsourced annotators, and expert annotators. Based on the predicted two-dimensional ambiguity scores, a dynamic scheduling strategy automatically routes data instances to the optimal annotation resources or resource combinations. For tasks with high cognitive ambiguity, an upgrade mechanism combining crowdsourced voting and expert arbitration is adopted to output the final annotated dataset. This invention can intelligently match annotation costs and capabilities based on the inherent attributes of the task, significantly reducing annotation costs and improving annotation efficiency while ensuring data annotation quality, thus solving the problems of high costs in traditional crowdsourcing and unstable annotation quality of single models.

[0008] To achieve the above objectives, the technical solution adopted by this invention is: a multi-level human-computer collaborative data annotation method based on task fuzziness evaluation, comprising the following steps:

[0009] S1. Construct and train the ambiguity prediction model: Randomly extract an exploration sample set from the original dataset, and simultaneously distribute each sample in the exploration sample set to multiple resources in the four-level annotation resource pool for exploratory annotation; Based on the exploratory annotation results, calculate the cognitive ambiguity ground truth label for each sample. And logical fuzziness truth labels ;in, Calculated using the information entropy formula Consistency and model stability calculations based on high-performance large language models and human consensus; fuzzy prediction models trained using machine learning algorithms. Establish sample feature vectors The mapping relationship to two-dimensional fuzzy labels, minimizing the loss function. Thus, a well-trained ambiguity prediction model is obtained;

[0010] S2. Ambiguity Prediction of Data to be Labeled: Feature vectors are extracted from the data to be labeled, and these feature vectors are input into the optimal ambiguity prediction model trained in step S1 to obtain the prediction score for each piece of data to be labeled. ;in, To score cognitive ambiguity, Score the logical ambiguity;

[0011] S3. Data routing allocation: Based on the cognitive ambiguity score obtained in step S2. And logical fuzziness score The data to be labeled is routed to the corresponding resources in the four-level labeling resource pool for processing; the four-level labeling resource pool includes low-performance large language models, high-performance large language models, ordinary crowdsourced labelers, and expert labelers.

[0012] S4. Generation of primary annotation results: Based on the resources allocated in step S3, each model is processed to obtain primary annotation results;

[0013] S5. Final labeled dataset output: Trigger the upgrade mechanism for the high ambiguity tasks generated during the routing process in steps S3-S4, obtain the final adjudication result, and aggregate all labeled results to output the final labeled dataset.

[0014] As an improvement of the present invention, in step S1, the truth label of cognitive ambiguity Based on the crowdsourced annotation results set The frequency of each tag category is calculated using the following method:

[0015]

[0016] In the formula, M represents the total number of label categories defined for the task. For the first Class tags in Frequency of appearance in the name taggers;

[0017] The logical fuzziness truth label The calculation method is as follows:

[0018]

[0019] In the formula, The mode of the crowdsourced annotation result set; The annotation results are for a high-performance large language model; This is an indicator function that takes the value 1 when the condition inside the parentheses is met, and 0 otherwise; This represents the number of times the most frequent label appears in the R-fold annotation of the low-performance model; and These are the weighting coefficients.

[0020] As another improvement of the present invention, the routing allocation principle in step S3 is as follows: setting a first threshold for cognitive ambiguity. Second threshold of cognitive ambiguity First threshold for logical ambiguity and the second threshold of logical ambiguity ,in, , ;

[0021] like If so, the data to be labeled will be routed to a low-performance large language model for automatic labeling;

[0022] like and Then the data to be labeled will be routed to a high-performance large language model for automatic labeling;

[0023] like If so, the data to be labeled will be routed to ordinary crowdsourced labelers for labeling;

[0024] like Then the data to be labeled will be routed to a high-performance large language model to generate pre-labeling and inference processes, and then transferred to expert annotators for verification;

[0025] like If so, the data to be labeled will be directly routed to expert labelers for arbitration labeling.

[0026] As another improvement to the present invention, in each model processing step S4,

[0027] If the data to be labeled is routed to a low-performance large language model or a high-performance large language model, the labeling results generated by the model are directly obtained as the primary labeling results.

[0028] If the data to be labeled is routed to a regular crowdsourced labeler, then obtain... The voting results of each labeler were tallied, and the label category with the most votes was determined. and the number of votes Calculate the confidence level of the vote ;like Consistency threshold ,but As a preliminary annotation result; if If so, the task is determined to be a high cognitive ambiguity task, triggering the upgrade mechanism;

[0029] If the data to be labeled is routed to a mode that requires verification, the labeling results that have been manually verified are used as the initial labeling results.

[0030] If the data to be labeled is routed to the expert arbitration mode, the final ruling label of the expert annotator will be used as the initial labeling result.

[0031] As another improvement of the present invention, the high ambiguity task in step S5 refers to the data to be labeled being routed to ordinary crowdsourced labelers, and the voting confidence level... Consistency threshold The task, in which voting confidence , The number of annotators, The number of votes for the tag category that received the most votes;

[0032] The upgrade mechanism specifically involves: combining high-fuzzy task samples with crowdsourced annotation distribution vectors. Pushed to the expert review queue, among which For the first The number of votes received by the class tag The total number of label categories; experts calculate the consistency coefficient. ,in The final confirmation label is submitted to experts through crowdsourcing. The number of votes; weighted aggregation of multi-source annotation results, and assignment of quality confidence scores to each annotated data point. For each labeled data, a structured labeled record is generated, which includes the original data, the final label, the label source, and the quality confidence score. All structured labeled records are then aggregated to form the final labeled dataset.

[0033] To achieve the above objectives, the present invention also adopts the following technical solution: a multi-level human-computer collaborative data annotation system based on task fuzziness evaluation, comprising a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the methods described above.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) The present invention provides a multi-level human-machine collaborative data annotation method based on task fuzziness evaluation, which introduces cognitive fuzziness. and logical ambiguity The two-dimensional quantitative evaluation formula is the first to achieve a refined and mathematical quantitative diagnosis of the difficulty of data labeling tasks, overcoming the one-sidedness of traditional methods that rely solely on model confidence to judge task difficulty.

[0036] (2) This invention constructs a four-level resource pool of large language models with different capabilities and costs and human annotators, and designs a dynamic routing strategy based on fuzziness score intervals, which can automate simple tasks, intelligentize logical tasks, crowdsource subjective tasks, and expertize difficult tasks. This "tailored" allocation method significantly reduces the overall annotation cost and greatly improves annotation efficiency while ensuring that the annotation quality is no lower than that of pure manual annotation.

[0037] (3) The present invention designs an exploratory training mechanism and a confidence level based on crowdsourced voting. The dynamic upgrade mechanism enables the system to start without relying on a large amount of prior labeled data and has the robustness to handle sudden high-difficulty samples, effectively solving the illusion problem and poor consistency problem that large language models have when dealing with long-tail and complex samples.

[0038] (4) The method of this invention constructs a feedback-based online learning closed loop. The "highly controversial samples" collected through the upgrade mechanism and their expert arbitration results are not only used to resolve the labeling disagreements of the current task, but are also transformed into high-quality training data for continuous iterative optimization of the ambiguity prediction model itself. This design enables the system to actively learn from the most difficult-to-label data, continuously enhancing its prediction accuracy for task ambiguity, thereby achieving self-evolution of the labeling strategy and continuous improvement of labeling efficiency. Attached Figure Description

[0039] Figure 1 This is a flowchart of the steps of the multi-level human-machine collaborative data annotation method based on task fuzziness evaluation of the present invention. Detailed Implementation

[0040] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0041] Example 1

[0042] A multi-level human-computer collaborative data annotation method based on task fuzziness assessment is proposed. This method quantifies the cognitive and logical fuzziness of the task and dynamically schedules a four-level annotation resource pool, including low-performance large language models, high-performance large language models, ordinary crowdsourced annotators, and expert annotators, to achieve an optimal balance between annotation cost and quality. Figure 1 As shown, the method specifically includes the following steps:

[0043] Step S1: Construct and train the ambiguity prediction model: The ambiguity prediction model is used to receive the feature vector of the data to be labeled and output the cognitive ambiguity score of the data. And logical fuzziness score .

[0044] S101, From the original dataset to be labeled In this study, a certain number of sample data were drawn to form an exploratory sample set using simple random sampling or stratified sampling methods. In this embodiment, the sample set is explored. Number of samples The preferred setting is 200 to 500 entries to ensure the representativeness of the sample distribution while reducing pre-labeling costs.

[0045] S102, Explore the sample set Each sample in Simultaneously, it is distributed to various representative resources in the four-level annotation resource pool for parallel annotation to obtain annotation feedback from different dimensions.

[0046] In this embodiment, the sample can be Distribute to K ordinary crowdsourced annotators to obtain a set of crowdsourced annotation results. At the same time, the sample Input a low-performance, large language model and set a high temperature parameter to stimulate randomness in generation; repeat the process. Next, obtain the low-performance model annotation result set. In addition, the samples Input a high-performance large language model and set the temperature parameter to 0 to obtain deterministic results, thus obtaining the high-performance model annotation results. .

[0047] S103. Based on the multi-source labeled data obtained in step S102, for each sample in the exploration sample set... Calculate the truth label of cognitive ambiguity And logical fuzziness truth labels Among them, the truth label of cognitive ambiguity Based on the crowdsourced annotation results set The frequency of each tag category was counted, and the information entropy was calculated using the following formula:

[0048]

[0049] In the formula, M represents the total number of label categories defined for the task. For the first Class tags in Frequency of appearance in the name taggers.

[0050] Logical fuzziness truth label Taking into account both the deviation between high-performance models and human consensus, as well as the inherent instability of low-performance models, the calculation formula is as follows:

[0051]

[0052] In the formula, This represents the mode of the crowdsourced annotation result set, i.e., the result of human consensus. The annotation results are for a high-performance large language model; This is an indicator function that takes the value 1 when the condition inside the parentheses is met, and 0 otherwise; This represents the number of times the most frequent label appears in the R-fold annotation of the low-performance model; and These are the preset weighting coefficients.

[0053] S104. Using a pre-trained deep learning model to process the samples Perform feature encoding to obtain a fixed-dimensional feature vector. If the data to be labeled is text data, the Sentence-BERT model can be used to extract semantic feature vectors; if the data to be labeled is image data, the output of the fully connected layer of a convolutional neural network model such as ResNet can be used as the visual feature vector.

[0054] The specific models mentioned above are merely examples, such as Sentence-BERT and ResNet. Those skilled in the art can use other feature extraction models or machine learning models that can achieve the same or similar functions to replace them according to the actual task requirements.

[0055] S105. Construct a lightweight fuzzy prediction model. , using the feature vector of the sample As input, the two-dimensional ambiguity truth label calculated in step S103 Supervised training is performed using the output target. The fuzzy prediction model F can be implemented using the gradient boosting decision tree algorithm. The loss function during training is defined as the mean squared error between the predicted and true values, aiming to minimize the following objective function:

[0056]

[0057] In the formula, and The model is paired with the sample. The predicted outputs for cognitive and logical ambiguity. During training, the sample set will be explored. The model is divided into a training set and a validation set. It is trained iteratively until the loss function converges, and finally a well-trained ambiguity prediction model is obtained.

[0058] Step S2: Extract features from the data to be labeled, and input the extracted feature vectors into the ambiguity prediction model to obtain the cognitive ambiguity score and logical ambiguity score for each piece of data to be labeled.

[0059] S201. Obtain the dataset to be labeled. This dataset contains a large number of raw data instances that have not yet been labeled. Traversing the dataset... Read each data instance to be labeled in sequence. .

[0060] S202, Examples of Data to be Annotated Feature extraction is performed to obtain feature vectors. In this step, the feature extraction method and deep learning model used must be strictly consistent with the feature extraction method used for training data in step S104 to ensure consistency of the feature space. Specifically, if the data is text, the same pre-trained Sentence-BERT model as in step S104 is used to extract semantic features; if the data is image, the same ResNet model as in step S104 is used to extract visual features.

[0061] S203, The feature vector obtained in step S202 is... Input the ambiguity prediction model trained in step S1 .Model Perform forward inference computation and output the results for this data instance. The two-dimensional ambiguity prediction result, i.e., the cognitive ambiguity score. And logical fuzziness score .

[0062]

[0063] In the formula, This indicates the degree to which the i-th data instance predicted by the prediction model causes subjective human disagreement. These scores represent the logical reasoning ability required for the predicted i-th data instance and the degree of factual uncertainty. The numerical range of these two scores is typically within the interval [0, 1], and they are used for subsequent dynamic routing decisions.

[0064] Step S3: Based on the cognitive ambiguity score and logical ambiguity score, and in conjunction with a preset dynamic scheduling strategy, the data to be labeled is routed to the corresponding resources in the four-level labeling resource pool for processing. The four-level labeling resource pool includes low-performance large language models, high-performance large language models, ordinary crowdsourced labelers, and expert labelers; the low-performance large language models are pre-trained language models with small parameter counts or low invocation costs; the high-performance large language models are pre-trained language models with large parameter counts, complex reasoning capabilities, and high invocation costs.

[0065] The preset dynamic scheduling strategy specifically involves: initializing the threshold parameters of the dynamic scheduling strategy; and setting a first threshold for cognitive ambiguity. and the second threshold of cognitive ambiguity ,in Set a first threshold for logical ambiguity. and the second threshold of logical ambiguity ,in These threshold parameters can be preset based on specific annotation budgets and quality requirements, or determined through small-sample testing.

[0066] Determine the data instance to be labeled Does it belong to a task with extremely low ambiguity?: If the cognitive ambiguity score obtained in step S203 is... And logical fuzziness score If the data instance is deemed a simple task, the system automatically routes it to a low-performance large language model. In this case, the system calls the API of the low-performance large language model to directly generate the annotation results without manual intervention.

[0067] Determine the data instance to be labeled Does it belong to a low logical ambiguity task? If the cognitive ambiguity score is... And the logical fuzziness score satisfies If the data instance is deemed to have relatively little subjective controversy, it requires strong logical reasoning ability, which exceeds the capabilities of the low-performance model. The system then routes it to a high-performance large language model, leveraging its superior reasoning capabilities for automatic annotation.

[0068] Determine the data instance to be labeled Does it belong to a task with moderate cognitive ambiguity? If the cognitive ambiguity score meets the following criteria: And logical fuzziness score If the data instance involves certain human common sense or subjective judgment, and the machine cannot accurately grasp it, the system will distribute it to the resource pool of ordinary crowdsourced annotators.

[0069] Determine the data instance to be labeled Does it belong to a high logical ambiguity task? If the logical ambiguity score is high... If the data instance is deemed to have extremely complex logic or requires rigorous fact-checking, then the system is determined to be flawed. The system adopts a collaborative "generate-verify" model: first, it routes to a high-performance large language model, requiring it to generate pre-annotated results and reasoning chains; then, the original data, pre-annotated results, and reasoning process are transferred to the expert annotator for rapid verification or correction.

[0070] Determine the data instance to be labeled Does it belong to a high cognitive ambiguity task? If the cognitive ambiguity score is high... If the data instance is deemed to have extremely high subjective controversy, the system will directly route it to an expert annotator for arbitration annotation. Alternatively, if either of the above conditions cannot be met, i.e., it falls into other complex ranges defined by the threshold, the system will default to expert arbitration mode to ensure the annotation quality of high-difficulty tasks.

[0071] Based on the above technical solution: by quantitatively distinguishing between cognitive ambiguity / subjective consensus difficulty and logical ambiguity / reasoning difficulty, the difficulty type of the task can be accurately identified, thereby avoiding the situation where simple tasks consume human resources and complex logical tasks are handled by weak models alone, and achieving optimal matching of heterogeneous annotation resources.

[0072] Step S4: Obtain the corresponding primary annotation results based on the different resource types to which the data to be annotated is routed.

[0073] If the data to be labeled is routed to a low-performance or high-performance large language model, the system receives the generated text returned by the model through an API interface. The system parses the generated text, extracts the structured label information, and directly uses it as the initial labeling result for that data instance.

[0074] If the data to be labeled is routed to regular crowdsourced labelers, the system collects the labeling results submitted by N regular labelers and performs consistency checks. Specifically, it counts the tag category with the most votes among all the labeling results. and the corresponding number of votes Then, the voting confidence level is calculated using the following formula. :

[0075]

[0076] The calculated voting confidence level Consistency threshold with preset Compare. If (For example = 0.6 (meaning a majority vote exceeding 60%), then the crowdsourcing result is considered valid, and the tag category with the highest number of votes is selected. This serves as the initial annotation result for this data instance. Based on the above technical solution: introducing a consistency detection formula based on voting proportion in the ordinary crowdsourcing process can dynamically discover potential high-difficulty subjective tasks that the prediction model fails to identify, and ensure data quality through an upgrade mechanism, thereby enhancing the robustness of the system.

[0077] like If the task is predicted to have moderate cognitive ambiguity, it is determined that the actual annotation process triggered unexpected human disagreement, classifying it as a potentially high cognitive ambiguity task. In this case, the system does not adopt the current crowdsourcing results but instead marks the data instance as a "highly controversial sample," triggering the upgrade mechanism in step S5.

[0078] If the data to be labeled is routed to a mode requiring verification, i.e., a high-logic-ambiguity task, the system displays the pre-labeling results and inference process generated by the high-performance large language model on the expert verification interface. Expert annotators review the pre-labeling results; if they approve, they confirm directly; if they disagree, they modify them. The system records the final labels submitted by the experts, using them as the initial labeling results for that data instance.

[0079] If the data to be labeled is routed to the expert arbitration mode, i.e., a task with high cognitive ambiguity, the system directly records the final ruling label given by the expert annotator as the initial labeling result for that data instance.

[0080] Step S5: Trigger the upgrade mechanism for specific high-ambiguity tasks generated during the routing process, obtain the final adjudication result, aggregate all annotation results, and output the final annotation dataset.

[0081] When determining the confidence level of the vote At that time, the system will use this data instance. This sample is marked as an anomaly. The system automatically extracts the crowdsourced annotation distribution vector for this sample. ,in Indicates the first The number of votes received for each category label, where M is the total number of categories. The system then processes the raw data. Together with the distribution vector Pushed to the expert review queue.

[0082] Expert labelers review the raw data and crowdsourced distribution, and provide the final labels. Simultaneously, the system calculates the "crowdsourcing-expert consistency coefficient" for this sample. Used for subsequent evaluation of crowdsourcing quality:

[0083]

[0084] In the formula, The final confirmation label is submitted to experts through crowdsourcing. The number of votes. The lower the value, the more misleading the sample is to ordinary people, and the higher its training value.

[0085] The system aggregates annotation results from different sources using a unified weighted average. To quantify the quality of the final dataset, the system assigns a quality confidence score to each labeled data point. :

[0086] For the automatic annotation results from large language models:

[0087]

[0088] For consistent results from regular crowdsourcing:

[0089]

[0090] For results from expert verification or arbitration:

[0091]

[0092] Each record in the final output dataset contains: {original data, final label, label source, quality confidence score}. }

[0093] The system iterates through all processed tasks, writes the formatted records to a database or file system, and constructs the final labeled dataset. For image or text classification tasks, the output format can be a JSONL file; for sequence labeling tasks, the output format can be a BIO format file.

[0094] To optimize the prediction model using the hard examples generated by the upgrade mechanism, the system collects all samples that trigger the upgrade. And its expert-verified labels, to construct an incremental training set The system calculates the true ambiguity bias of these samples and uses this bias as weights to fine-tune and update the model. The loss function during the update is... Difficult example weight term introduced :

[0095]

[0096] in, .

[0097] This formula shows that the greater the deviation between crowdsourcing results and expert decisions, the more likely it is that... The smaller the value, the higher the sample weight. The larger the sample size, the more the model will focus on these challenging samples that are prone to prediction errors when updating. Through this step, the system achieves continuous evolution in its ambiguity prediction capabilities.

[0098] In summary, the method of this invention utilizes a lightweight fuzziness prediction model to predict the cognitive fuzziness score and logical fuzziness score of each data instance. It constructs a four-level annotation resource pool comprising low-performance large language models, high-performance large language models, ordinary crowdsourced annotators, and expert annotators. Based on the predicted two-dimensional fuzziness scores, a dynamic scheduling strategy automatically routes data instances to the optimal annotation resources or resource combinations. For tasks with high cognitive fuzziness, an upgrade mechanism combining crowdsourced voting and expert arbitration is adopted. While ensuring data annotation quality, this method significantly reduces annotation costs and improves annotation efficiency, maximizing annotation quality and efficiency within a limited budget.

[0099] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A multi-level human-machine collaborative data annotation method based on task fuzziness evaluation, characterized in that, Includes the following steps: S1. Construct and train the ambiguity prediction model: The ambiguity prediction model is used to receive sample data, taking the feature vector of the sample data as input and the two-dimensional ambiguity truth label as input. The model is used as the output target for supervised training, and the training is iterative until convergence is achieved, resulting in the best-trained fuzziness prediction model. For the truth label of cognitive ambiguity, For logical ambiguity truth labels; S2. Ambiguity Prediction of Data to be Labeled: Feature vectors are extracted from the data to be labeled, and these feature vectors are input into the optimal ambiguity prediction model trained in step S1 to obtain the prediction score for each piece of data to be labeled. ;in, To score cognitive ambiguity, Score the logical ambiguity; S3. Data routing allocation: Based on the cognitive ambiguity score obtained in step S2. And logical fuzziness score The data to be labeled is routed to the corresponding resources in the four-level labeling resource pool for processing; the four-level labeling resource pool includes low-performance large language models, high-performance large language models, ordinary crowdsourced labelers, and expert labelers. S4. Generation of primary annotation results: Based on the resources allocated in step S3, each model is processed to obtain primary annotation results; S5. Final labeled dataset output: Trigger the upgrade mechanism for the high ambiguity tasks generated during the routing process in steps S3-S4, obtain the final adjudication result, and aggregate all labeled results to output the final labeled dataset.

2. The multi-level human-machine collaborative data annotation method based on task fuzziness evaluation as described in claim 1, characterized in that: In step S1, the cognitive ambiguity truth label of the j-th element in the sample dataset. Based on the crowdsourced annotation results set The frequency of each tag category is calculated using the following method: ； In the formula, M represents the total number of label categories defined for the task. For the first Class tags in Frequency of appearance in the name taggers; Logical fuzziness truth label of the j-th element in the sample dataset The calculation method is as follows: ； In the formula, The mode of the set of crowdsourced annotation results; The annotation results are for a high-performance large language model; This is an indicator function that takes the value 1 when the condition inside the parentheses is met, and 0 otherwise; This represents the number of times the most frequent label appears in the R-fold annotation of the low-performance model; and These are the weighting coefficients.

3. The multi-level human-machine collaborative data annotation method based on task fuzziness evaluation as described in claim 2, characterized in that: The objective function for training the ambiguity prediction model in step S1 is specifically: ； In the formula, and The model is paired with the sample. The predicted outputs of cognitive ambiguity and logical ambiguity. The size of the training set for training the ambiguity predictor. This represents the j-th element in the dataset.

4. The multi-level human-machine collaborative data annotation method based on task fuzziness evaluation as described in claim 1, characterized in that: In step S2, the data to be labeled is the first... Cognitive ambiguity score of each element This indicates the degree to which the predictive model judges the data instance to cause subjective human disagreement; the first data instance in the unlabeled data... Logical fuzziness score of each element This indicates how the predictive model assesses the logical reasoning ability required for the data instance and the degree of factual uncertainty, specifically: ； in, This is a fuzzy prediction model.

5. The multi-level human-machine collaborative data annotation method based on task fuzziness evaluation as described in claim 1, characterized in that: The routing allocation principle in step S3 is as follows: A first threshold for cognitive ambiguity is set. Second threshold of cognitive ambiguity First threshold for logical ambiguity and the second threshold of logical ambiguity ,in, , ; like If so, the data to be labeled will be routed to a low-performance large language model for automatic labeling; like and Then the data to be labeled will be routed to a high-performance large language model for automatic labeling; like If so, the data to be labeled will be routed to ordinary crowdsourced labelers for labeling; like Then the data to be labeled will be routed to a high-performance large language model to generate pre-labeling and inference processes, and then transferred to expert annotators for verification; like If so, the data to be labeled will be directly routed to expert labelers for arbitration labeling.

6. The multi-level human-machine collaborative data annotation method based on task fuzziness evaluation as described in claim 5, characterized in that: In each model processing step S4, If the data to be labeled is routed to a low-performance large language model or a high-performance large language model, the labeling results generated by the model are directly obtained as the primary labeling results. If the data to be labeled is routed to a regular crowdsourced labeler, then obtain... The voting results of each labeler were tallied, and the label category with the most votes was determined. and the number of votes Calculate the confidence level of the vote ;like Consistency threshold ,but As a preliminary annotation result; if If so, the task is determined to be a high cognitive ambiguity task, triggering the upgrade mechanism; If the data to be labeled is routed to a mode that requires verification, the labeling results that have been manually verified are used as the initial labeling results. If the data to be labeled is routed to the expert arbitration mode, the final ruling label of the expert annotator will be used as the initial labeling result.

7. The multi-level human-machine collaborative data annotation method based on task fuzziness evaluation as described in claim 1, characterized in that: The high-ambiguity task in step S5 refers to the data to be labeled being routed to ordinary crowdsourced labelers, and the voting confidence level... Consistency threshold The task, in which voting confidence , The number of annotators. The number of votes for the tag category that received the most votes; The upgrade mechanism specifically involves: combining high-fuzzy task samples with crowdsourced annotation distribution vectors. Pushed to the expert review queue, among which For the first The number of votes obtained by the class tag The total number of label categories; experts calculate the consistency coefficient. ,in The final confirmation label is submitted to experts through crowdsourcing. The number of votes; weighted aggregation of multi-source annotation results, and assignment of quality confidence scores to each annotated data point. Construct the final labeled dataset.

8. The multi-level human-machine collaborative data annotation method based on task fuzziness evaluation as described in claim 7, characterized in that: The quality confidence score in the upgrade mechanism of step S5 If the annotation results come from a large language model, then If the results are consistent with those from regular crowdsourcing, then ; If it comes from expert verification or arbitration, then .

9. The multi-level human-machine collaborative data annotation method based on task fuzziness evaluation as described in claim 1, characterized in that: In step S2, when extracting features from the data to be labeled, if the data to be labeled is text data, the Sentence-BERT model is used to extract semantic feature vectors; if the data to be labeled is image data, the ResNet model is used to extract visual feature vectors.

10. A multi-level human-computer collaborative data annotation system based on task fuzziness evaluation, comprising a computer program, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-9 above.

Images