Agent-based data labeling method, device and equipment, and storage medium

Abstract

The disclosure provides an agent-based data labeling method and device, equipment and a storage medium, relating to the technical field of computers, and in particular to the technical field of artificial intelligence such as large models, agents, multi-agents, model training, sample collection, automatic labeling, intelligent labeling. The specific implementation scheme is: based on the classification result of the sample data by the main model and one or more auxiliary models, pre-screening the candidate class set of the sample data, the candidate class set including a plurality of candidate classes; through the expert classification agent corresponding to each candidate class, respectively classifying the sample data to obtain the respective classification results; according to the classification results of the expert classification agent corresponding to each candidate class, a decision is made to obtain the labeling result of the sample data.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence technology such as large models, intelligent agents, multi-agent systems, model training, sample collection, automatic annotation, and intelligent annotation. Background Technology

[0002] With the rapid development of large language models and deep learning technologies, high-quality labeled data has become a bottleneck for model training and optimization. The scale and quality of labeled data determine the performance ceiling of downstream models. Purely manual annotation offers high quality but is also costly, and annotation results from a single annotator's perspective have limitations. Summary of the Invention

[0003] This disclosure provides a data annotation method, apparatus, device, and storage medium based on intelligent agents.

[0004] According to one aspect of this disclosure, an agent-based data annotation method is provided, comprising: Based on the classification results of the sample data by the main model and one or more auxiliary models, a candidate category set of the sample data is pre-screened, and the candidate category set includes multiple candidate categories; The sample data is classified by an expert classification agent corresponding to each candidate category, and the classification results are obtained respectively. The labeling results of the sample data are obtained by making decisions based on the classification results of the expert classification agents corresponding to each candidate category.

[0005] According to another aspect of this disclosure, an agent-based data annotation apparatus is provided, comprising: The candidate generation module is used to pre-screen the candidate category set of the sample data based on the classification results of the main model and one or more auxiliary models. The candidate category set includes multiple candidate categories. The classification agent module is used to classify the sample data by using expert classification agents corresponding to each candidate category, and obtain their respective classification results; The decision module is used to make decisions based on the classification results of the expert classification agents corresponding to each candidate category, and to obtain the labeling results of the sample data.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating a data annotation method based on an intelligent agent according to an embodiment of the present disclosure; Figure 2 This is a flowchart illustrating a data annotation method based on an intelligent agent according to an embodiment of the present disclosure; Figure 3 This is a flowchart illustrating a data annotation method based on an intelligent agent according to an embodiment of the present disclosure; Figure 4 This is a flowchart illustrating a data annotation method based on an intelligent agent according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the structure of a self-evolving data annotation method and system based on multi-agent collaboration according to an embodiment of this disclosure; Figure 6 This is a flowchart illustrating a multi-agent self-evolutionary data annotation method according to an embodiment of the present disclosure; Figure 7 This is a schematic diagram of the structure of an agent-based data annotation device according to an embodiment of the present disclosure; Figure 8 This is a schematic diagram of the structure of an agent-based data annotation device according to an embodiment of the present disclosure; Figure 9 This is a block diagram of an electronic device used to implement embodiments of the present disclosure. Detailed Implementation

[0011] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0012] In real-world business scenarios, data annotation faces multi-dimensional technical challenges. On the one hand, while traditional manual annotation can guarantee a certain level of annotation quality, it is costly, time-consuming, and limited by differences in annotators' understanding of complex annotation specifications, making it difficult to ensure consistency among different annotators. Especially in complex annotation tasks involving dozens or even hundreds of label categories, annotators need to master a large number of classification definitions, boundary conditions, and decision rules simultaneously, resulting in extremely high cognitive load and a rising error rate. On the other hand, while automatic annotation methods based on large language models significantly reduce labor costs, the lack of task-specific annotation experience and precise domain knowledge of large language models (LLMs) leads to unstable decision quality in semantically ambiguous boundary cases, and the need to perform a complete LLM inference call for each sample results in huge computational resource consumption.

[0013] Specifically, current data annotation technologies face the following problems: (1) The contradiction between annotation quality and cost: Pure manual annotation has controllable quality but the cost increases linearly with the data scale. Pure LLM annotation has a lower cost but the error rate increases significantly in boundary cases and complex multi-label scenarios. There is a lack of a mechanism that can adaptively allocate different annotation resources to samples of different difficulty.

[0014] (2) Limitations of a single annotation perspective: Whether it is manual or LLM annotation, each sample is usually judged by a single annotator, lacking multi-perspective cross-validation and conflict resolution mechanisms. When the sample involves semantically overlapping areas of multiple possible labels, a single perspective is prone to systematic bias.

[0015] (3) Static solidification of the annotation system: The judgment rules of the annotation system are usually solidified after deployment, and cannot automatically accumulate experience, discover patterns and evolve from the annotation process. The boundary cases and difficult judgment patterns that frequently occur in the annotation are not systematically captured and utilized, resulting in the repeated occurrence of the same type of error.

[0016] (4) Task binding of annotation framework: Annotation systems are usually designed specifically for specific task types (such as text classification or sequence labeling), and the system needs to be rebuilt when switching to a new annotation task. There is a lack of a task-independent general annotation framework that can adapt to different types of annotation needs through configuration.

[0017] The following methods can improve the quality and efficiency of data annotation: 1. Crowdsourced Labeling Pipeline: This method achieves labeling quality control through multi-person labeling combined with majority voting or expert review. For example, 3-5 labelers are assigned to each sample, and the majority consensus is used as the final label, while dissenting samples are decided by experts. This method can provide high labeling consistency, but the labor cost is proportional to the data scale, and the training and consistency maintenance costs for labelers are extremely high in complex labeling systems.

[0018] Second: Automatic annotation based on a single large language model. In recent years, researchers have explored using large language models to directly annotate data. By carefully designing prompts, annotation specifications and examples are embedded in the prompt text to guide the LLM to output annotation results. Some methods introduce a chain-of-thought mechanism, allowing the LLM to perform reasoning before providing labels, thereby improving the quality of boundary case judgments. This method reduces human labor costs, but requires a complete LLM reasoning call for each sample, resulting in huge computational overhead, and the LLM lacks a multi-perspective verification mechanism in a single judgment.

[0019] 3. Annotation Optimization Based on Active Learning. Active learning methods prioritize the most informative samples for manual annotation by estimating the model's uncertainty, thereby maximizing model performance within a limited annotation budget. Examples include entropy-based uncertainty sampling and query-by-committee based queries. This method effectively reduces redundant annotations, but still relies on manual annotation, and the accuracy of the uncertainty estimation itself is limited by the capabilities of the current model.

[0020] 4. Voting-based labeling using multi-model ensembles. This approach trains multiple classifiers or invokes multiple LLMs to independently label the same sample, then merges the results through a voting mechanism. Representative methods include label aggregation based on the Dawid-Skene model, and the Generative Model of Labels, Abilities, and Difficulties (GLAD) algorithm. This method improves labeling robustness through a multi-model perspective, but all models perform complete inference for all samples, lacking an adaptive resource allocation mechanism based on sample difficulty.

[0021] The above methods still have one or more problems, such as balancing annotation quality and cost, resolving conflicts from multiple perspectives, system self-evolution, and task generality.

[0022] Figure 1 This is a flowchart illustrating an agent-based data annotation method 100 according to an embodiment of the present disclosure. In one implementation, the method may include: S110. Based on the classification results of the sample data by the main model and one or more auxiliary models, pre-screen the candidate category set of the sample data, wherein the candidate category set includes multiple candidate categories; S120. The sample data is classified by the expert classification agent corresponding to each candidate category to obtain the respective classification results; S130. Make a decision based on the classification results of the expert classification agent corresponding to each candidate category to obtain the labeling results of the sample data.

[0023] The agent-based data annotation method of this disclosure can be executed by a multi-agent collaborative system, also known as an agent-based data annotation system. Multi-agent collaborative frameworks exhibit superior performance in areas such as code generation and inference tasks. By decomposing complex tasks into sub-tasks and assigning them to agents with different roles to complete collaboratively, performance exceeding that of a single agent can be achieved. This disclosure, through a multi-agent system in a data annotation scenario, executes a data annotation method tailored to the characteristics of the annotation task (such as labeling systems, confidence assessments, and experience accumulation).

[0024] In this embodiment, the system may include a central controller, such as an orchestration module, for interacting with other modules or agents within modules, and pre-configuring the scheduling order and dynamic jump methods of each module or agent. The system may include a pre-screening candidate generation module, which includes one or more of a main model adapter, auxiliary model adapters, and a candidate set generator. The main model adapter calls the main model to classify the sample data to be labeled, obtaining multiple classification results. The auxiliary model adapter calls one or more auxiliary models in parallel to classify the sample data, with each auxiliary model obtaining multiple classification results. The candidate set generator pre-screens the candidate category set of the sample data based on the classification results of the main model and each auxiliary model. The system may include a classification agent pool, which can schedule expert classification agents corresponding to each candidate category in the candidate category set in parallel to classify the sample data and obtain their respective classification results. The system may include a decision module, which can combine the classification results of different expert classification agents to make a decision and obtain the labeling results of the sample data.

[0025] According to embodiments of this disclosure, different expert classification agents are scheduled to perform classification based on the classification results of multiple models. This can improve the labeling quality through multiple perspectives, and improve labeling efficiency through parallel classification and automatic labeling by a multi-agent system.

[0026] In one implementation, S110 pre-screens the candidate category set of the sample data based on the classification results of the main model and one or more auxiliary models, which may include: adding one or more classification results of the main model to the candidate category set according to the differences between multiple classification results of the main model; and adding one or more classification results of each of the one or more auxiliary models to the candidate category set.

[0027] In this embodiment, the pre-screening candidate generation module in the system may further include one or more of a consistency judge and a threshold judge. After the main model adapter calls the main model, it can return partial classification results of the main model, such as the category IDs, category names, and probabilities of Top-1 and Top-2. After the auxiliary model adapter calls one or more auxiliary models in parallel, it can uniformly return partial classification results of each auxiliary model, such as the Top-1 result. The consistency judge checks whether the classification results of all models are consistent, for example, whether the first-ranked category identifier (Top-1 ID) of all models is consistent. The threshold judge can calculate the differences between different classification results of the main model. For example, the threshold judge can calculate the probability difference between Top-1 and Top-2 of the main model and determine whether Top-2 should enter the candidate set. If the probability difference is large, Top-2 may not enter the candidate set; otherwise, it may enter the candidate set. The candidate set generator summarizes the candidate categories of the main model and the specified categories of the auxiliary models, such as Top-1.

[0028] According to embodiments of this disclosure, by comparing the differences between multiple classification results of the main model, suitable categories can be added to the candidate set. By adding partial classification results of each auxiliary model to the candidate category set, the diversity of candidate categories can be improved, multi-view annotation can be achieved, and the annotation quality can be improved.

[0029] In one implementation, S110 pre-screens the candidate category set of the sample data based on the classification results of the main model and one or more auxiliary models, and may further include: adding necessary categories corresponding to the classification scenario to the candidate category set; and performing deduplication processing on each category in the candidate category set.

[0030] In this embodiment, the candidate set generator, in addition to summarizing the candidate categories of the main model and some categories of the auxiliary model, can also add the IDs of necessary categories corresponding to the classification scenario to the candidate category set. For example, the necessary category in the risk identification scenario can be the "normal" category, and its ID can be 0. Similarly, the necessary category in the sentiment analysis scenario can be the "neutral" or "normal" category, etc. Since the output results and necessary categories of different models may overlap, deduplication can be performed on all candidate categories in the candidate category set, or the candidate set can be added only if there are no duplicates.

[0031] According to embodiments of this disclosure, by adding necessary categories corresponding to the classification scenario to the candidate category set, necessary categories can participate in subsequent competition instead of being simply discarded; deduplication can reduce redundant categories and improve the efficiency of subsequent classification and labeling.

[0032] Figure 2 This is a flowchart illustrating an agent-based data annotation method 200 according to an embodiment of the present disclosure. This method can be used to implement one or more features of the aforementioned agent-based data annotation method. In one embodiment, the method may further include: S210. A preliminary classification agent is awakened to perform a preliminary classification of the sample data, and the preliminary classification result is obtained. The preliminary classification result is used to indicate whether the sample data needs to be classified by the classification agent pool.

[0033] In one implementation, the triggering method for waking up the preliminary classification agent includes one or more of the following: A consistency judgment is made on the classification results of the main model and the one or more auxiliary models to obtain a consistency result; It is triggered in parallel with the function of filtering the set of candidate categories.

[0034] In this embodiment, the system may include a preliminary classification agent, the name of which may vary depending on the scenario. For example, in a risk identification scenario, the preliminary classification agent may be a risk agent, used to perform binary classification on sample data to determine whether the sample data is risky or not. If risky, the sample data needs to be further refined through a pool of classification agents; if not risky, it can skip further processing. Similarly, in a sentiment analysis scenario, the preliminary classification agent may be a sentiment agent, used to perform binary classification on sample data to determine whether the sample data is sentimental or not. If sentimental, the sample data needs to be further refined through a pool of classification agents; if not sentimental, it can skip further processing. Other classification scenarios are similar and will not be exhaustively listed here.

[0035] In this embodiment, there are several ways to trigger the initial classification agent. It can be triggered in parallel or asynchronously with the pre-screening candidate generation module's function of filtering the candidate category set. Alternatively, it can be triggered based on the consistency judgment result of the consistency judge. For example, if the consistency judge detects that the top-1 category identifiers (Top-1IDs) of all models are consistent (e.g., all are "normal" IDs), the initial classification agent can be triggered to perform further binary classification. The jump path is determined based on the binary classification result. For example, in a risk identification scenario, if the initial classification agent determines there is risk, early exit can be prohibited, and the process can be forced into the subsequent routing and expert judgment stages. If the initial classification agent determines there is no risk, the labeling result of the sample data can be determined as "normal," and the subsequent process can be exited.

[0036] According to embodiments of this disclosure, by using a preliminary classification agent to perform coarse classification of sample data, early exit can be reasonably prevented, and data that does not need to be finely classified by the classification agent pool can be filtered out, thereby reducing the amount of data that needs to be processed subsequently and improving classification and labeling efficiency.

[0037] In one embodiment, the method may further include: S220. Receive the sample data, the candidate category set, and the complete category set through the routing agent; S230. Select a portion of the categories from the complete category set and add them to the candidate category set.

[0038] In this embodiment, based on the jump path determined by the initial classification agent, if further classification is needed, execution can jump to the routing agent to continue. The routing agent receives sample data and candidate category sets from previous modules. It can also load the complete category set, which can be stored in the routing agent, the scheduling module (obtained during initial interactions), or other modules; the specific configuration can be flexibly adjusted according to requirements. From the complete category set, a subset of categories, such as the top three (Top-3), can be selected and added to the candidate category set.

[0039] According to embodiments of this disclosure, by cooperating with the routing agent and the preliminary classification agent, path switching and data transfer can be completed quickly; by adding selected categories from the complete category set to the candidate category set, the number of categories participating in the competition can be expanded, preventing the pre-screening model from missing any categories.

[0040] Figure 3This is a flowchart illustrating an agent-based data annotation method 300 according to an embodiment of the present disclosure. This method can be used to implement one or more features of the aforementioned agent-based data annotation method. In one implementation, S120 uses expert classification agents corresponding to each candidate category to classify the sample data respectively, obtaining respective classification results, including: S310. Based on the classification agent pool, instantiate the corresponding expert classification agent for each candidate category in the candidate category set; S320. Assemble prompt text for the expert classification agent corresponding to each candidate category. The prompt text includes one or more of the following: the meaning of its own candidate category, the judgment rule, the set of positive examples, the set of negative examples, the boundary cases, the historical cases, and the induction rule. S330. The expert classification agents corresponding to each candidate category asynchronously and in parallel classify the sample data based on the prompt text to obtain their respective classification results.

[0041] In this embodiment, a classification agent pool can be set up, including an agent factory, a knowledge injector, a parallel executor, and a result aggregator. The agent factory dynamically instantiates expert classification agents based on a set of candidate categories, and each expert classification agent can be bound to a complete knowledge context of a candidate category. The knowledge injector assembles prompt text for each expert classification agent, including category definitions, judgment rules, positive and negative examples, boundary cases, and historical cases and inductive rules extracted from the system's experience management module, and renders them through a template engine. The parallel executor drives all expert classification agents to execute asynchronously and in parallel, without interdependence. The failure of a single expert classification agent does not block other expert classification agents. The result aggregator collects the positive or negative judgment results and confidence levels of all expert classification agents, counts the number of positive results, and determines the subsequent branch direction. According to this embodiment, by scheduling multiple expert classification agents in parallel, multi-perspective independent judgments can be provided, reducing systematic bias. Each expert classification agent carries exclusive knowledge and experience data for its corresponding category, providing more accurate and consistent annotation results.

[0042] In one implementation, S130 makes a decision based on the classification results of the expert classification agent corresponding to each candidate category to obtain the labeling results of the sample data, including one or more of the following steps: When only one expert classification agent outputs a positive result, the labeling result classification of the sample data is obtained according to the candidate category corresponding to the expert classification agent that outputs the positive result; If no expert classification agent in the candidate category set outputs a positive result, a complete category set is provided to the judgment agent, which is used to obtain the labeling result of the sample data based on the complete category set. When multiple expert classification agents output positive results, the judgment agent makes a decision by combining the classification results and reasoning basis of the expert classification agents corresponding to each candidate category, and obtains the labeling result of the sample data.

[0043] In this embodiment, the judging agent can make a comprehensive judgment based on the results output by various expert classification agents. When only one expert classification agent outputs a positive result, the judging agent can directly determine that category. When no expert classification agent outputs a positive result, the judging agent may not directly determine a specific necessary category, such as "normal," but instead receive the complete category set from the routing agent as a fallback and reselect the most suitable category. When multiple expert classification agents output positive results, the judging agent can combine the classification results and reasoning of each expert classification agent to make further decisions, thereby determining the final labeling result of the sample data.

[0044] According to the embodiments of this disclosure, by using the multi-perspective independent judgment of the parallel expert classification agent and the conflict resolution mechanism of the judgment agent, the systematic bias caused by a single annotation perspective is effectively avoided. The judgment agent makes refined decisions based on full-category rejection, hierarchical priority, etc., which can provide more accurate and consistent annotation results in boundary cases, candidate missed recalls and multi-label semantic overlap scenarios.

[0045] In one implementation, the labeling result of the sample data is obtained by comprehensively outputting the classification results and reasoning basis of the expert classification agents corresponding to each candidate category by the judgment agent, including: The confidence level is calibrated based on one or more of the following: the degree of consensus among the expert classification agents corresponding to each candidate category, the probability in the classification results of the main model, and historical retry information. The labeling result of the sample data is determined based on the confidence level of the judgment.

[0046] In this embodiment, the system may include a decision-making module, which may include an evidence aggregation submodule, a full-category rollback submodule, a hierarchical priority submodule, a confidence calibrator, and a divergence recorder. The evidence aggregation submodule presents the judgment results of the expert classification agent in a structured manner. When the number of positive results is 0, the full-category rollback submodule provides the judgment agent with a complete list of categories, enabling it to select the most appropriate risk category from all categories, and only outputs "normal" when it is certain that the sample is normal. The hierarchical priority submodule maintains the hierarchical relationship of the label system, prioritizing the adoption of finer-grained labels when conflicting labels have parent-child relationships. The confidence calibrator calibrates the final judgment confidence based on the consistency of the expert classification agent's opinions. The divergence recorder records minority opinions for the experience learning module to analyze label boundaries. The decision-making module takes as input the expert classification agent's judgment results, the full category list, and historical retry records, and outputs the final label, judgment confidence, divergence information, and manual review mark.

[0047] In some examples, the confidence level of a judgment can be calibrated based on the degree of consensus among expert classifiers, the probability of the master model, and historical retry information. For instance, if the confidence level is greater than a first threshold, it means that all positive expert classifiers agree on the same classification label, and this label can be used as the labeling result. If the confidence level is between the first and second thresholds, it means that there is majority agreement but minority disagreement among expert classifiers; the majority agreement result can be used, or a new round can be executed. If the confidence level is less than the second threshold, it means that there is significant disagreement among expert classifiers; no result is output for this round, but the historical record is carried over to the next round.

[0048] According to embodiments of this disclosure, the determination of the intelligent agent based on confidence calibration for refined decision-making can improve the accuracy of the annotation results.

[0049] In one implementation, determining the labeling result of the sample data based on the confidence level includes one or more of the following steps: If the confidence level is less than the confidence threshold, the candidate category set for the current round, the classification result of the expert classification agent, the judgment result, and the reason for the low confidence level are added to the history. If the maximum number of retries has not been reached, the agent-based sample data annotation method will be re-executed; If the maximum number of retries is reached, output the current best result and add a mark indicating that the sample data needs to be reviewed.

[0050] In this embodiment, if the confidence level is less than a confidence threshold (e.g., a second threshold), it indicates insufficient confidence. The judging agent can append the candidate category set of the current round, the classification result of the expert classification agent, the judgment result, and the reason for low confidence to the historical record, and then return to the pre-screening stage (see S110) with the complete historical record to retry until the confidence threshold or the maximum number of retries is reached. If the maximum number of retries is reached, the current best annotation result of the sample data is output, and this sample data is marked as needing review.

[0051] According to embodiments of this disclosure, historical records can provide experience for subsequent retry rounds, improving the accuracy of identification results in subsequent rounds.

[0052] Figure 4 This is a flowchart illustrating an agent-based data annotation method 400 according to an embodiment of the present disclosure. This method can be used to implement one or more features of the agent-based data annotation method described above. In one embodiment, the method may further include one or more of the following steps: S410. If the probability of the first classification result of the main model is greater than the confidence threshold, but the first classification result is inconsistent with the labeling result, add a verification mark to the sample data. S420. Add the sample data with verification marks and their annotation results to the case library.

[0053] In this embodiment, the high-confidence validator can perform post-validation on the final annotation result output by the judgment agent. For example, it compares whether the first-ranked category given by the main model with high confidence is consistent with the final annotation result. If they are inconsistent, the system can force manual review (add a mark indicating that review is required). The sample data of a certain category with the mark indicating that review is required and their annotation results are added to the case library, which can serve as negative examples for that category.

[0054] In one embodiment, the method may further include: S430. Push the sample data that meets the difficulty conditions from the sample data with the need for review to the manual review queue; wherein, the difficulty conditions include one or more of the following: the confidence level is less than the difficulty threshold; the ratio of the positive results to the negative results of the expert classification agent is less than the proportion threshold; the number of retries reaches the upper limit; the classification result of the preliminary classification agent conflicts with the pre-screening conclusion; the post-validation of the main model fails. S440. Add the received manual review results to the case library.

[0055] In this embodiment, the difficult sample identifier can further select difficult samples from the sample data with verification markers and push these difficult samples to the manual review queue. The manual review queue sorts the difficult samples according to judgment confidence and verification risk, and includes a complete multi-stage judgment process record for manual reference. The feedback loop receives the manual review results and can return the manual review results as high-confidence cases to the case library.

[0056] According to embodiments of this disclosure, the burden of manual review can be reduced and the targeting of manual review can be improved by reviewing and marking, and screening difficult samples. A two-layer experience-based self-evolution mechanism enables the annotation system to automatically accumulate experience from each annotation process, continuously enriching the reference case library for each label, and continuously improving judgment capabilities. The case library can be reused across tasks.

[0057] In one implementation, the method may further include: summarizing cases in the case library according to a specified category to obtain judgment rules and boundary conditions, so as to generate a set of structured rules corresponding to the specified category, wherein the set of structured rules corresponding to the specified category is used to instantiate the expert classification agent corresponding to the specified category.

[0058] In this embodiment, the case library can store labeled cases in partitions according to tags (categories), and a first-in-first-out (FIFO) strategy is used to control the capacity limit. Cases generated by manual review can be marked as high priority. When the number of cases for a certain tag reaches a preset threshold, the rule summarization engine calls the large language model to analyze the case set and automatically generates a structured rule set containing judgment rules and boundary conditions. The task configurator maintains parameters such as the tag system, model registry, preliminary classification / routing switch, expert classification agent template, main model screening threshold (e.g., top-2 threshold), judgment confidence threshold, maximum number of retries, and main model high confidence verification threshold.

[0059] According to embodiments of this disclosure, a continuous self-evolving closed loop is formed through case accumulation, rule summarization, and manual feedback, thereby improving judgment capabilities.

[0060] To address some issues in data annotation scenarios, this disclosure provides a data annotation method and system based on intelligent agents.

[0061] First, some data annotation methods lack an adaptive resource allocation mechanism based on sample difficulty. Whether it's crowdsourcing or LLM-based automatic annotation, the same annotation process and computational resources are applied to all samples. However, in real-world data distributions, a large number of samples are high-confidence "simple samples," requiring only lightweight judgment to provide accurate labels, while the truly in-depth "boundary samples" are only a minority. Investing the same amount of annotation resources in simple samples as in difficult samples results in serious resource waste. The data annotation method and system of this disclosure can establish an efficient pre-screening and stage-jumping mechanism, dynamically adjusting the annotation strategy according to the judgment difficulty of the samples.

[0062] Secondly, judgment results from a single annotation perspective lack reliable quality assurance. Annotation methods based on a single LLM rely on one-time inference to complete the judgment; when a sample falls into the semantically overlapping region of multiple labels, a single perspective struggles to detect potential misjudgments. Although multi-model ensemble voting methods introduce multiple perspectives, the judgments of each model are homogeneous global classifications, rather than specialized in-depth analysis for each candidate label. Furthermore, when conflicts arise between multiple perspectives, simple majority voting mechanisms cannot provide accurate results. The data annotation method and system of this disclosure can employ multi-perspective cross-validation to discover potential misjudgments, effectively utilizing label hierarchy relationships and confidence differences for refined arbitration.

[0063] Furthermore, some annotation systems lack the ability to automatically learn and evolve from the annotation process. A large amount of judgment results, reasoning, and manual correction information generated during annotation are not systematically captured and utilized. The experience accumulated by annotators in practice, such as "which judgment factors should be prioritized for a certain type of boundary case," remains at the individual level and cannot be distilled into system-level knowledge. Meanwhile, although active learning methods can identify high-information samples, they are typically only used for selecting model training data. The data annotation method and system of this disclosure can feed back the results of manual review to form a closed loop of experience accumulation and rule evolution within the annotation system itself.

[0064] Furthermore, the architecture of some annotation systems is tightly coupled to specific task types. For example, an annotation system designed for text classification cannot be directly used for sequence labeling or relation extraction tasks, and vice versa. Each time a new annotation requirement arises, the annotation process, annotation interface, and quality control mechanism need to be redesigned. The data annotation method and system of this disclosure can provide a task-independent general annotation meta-framework that can be adapted to different types of annotation tasks through standardized interface protocols and pluggable configuration.

[0065] The agent-based data annotation method and system proposed in this disclosure include a self-evolving data annotation method and system based on multi-agent collaboration, capable of achieving one or more objectives such as adaptive stratification of sample difficulty, multi-agent collaborative judgment and conflict resolution, automatic accumulation and evolution of annotation experience, and support for task-independent configuration. For example, sample difficulty stratification is achieved through pre-screening candidate generation via a pre-screening module; missed and recalled samples are reduced through initial screening routing (e.g., risk routing (Risk / Router)); multi-agent parallel judgment and arbitration are used to ensure annotation quality from multiple perspectives; continuous system optimization is achieved through post-verification and active learning-driven experience self-evolution; and the annotation framework is generalized through a pluggable task configuration protocol.

[0066] Figure 5 This is a schematic diagram of the structure of a self-evolving data annotation method and system based on multi-agent collaboration, according to an embodiment of this disclosure. By constructing a hierarchical pipeline of "pre-screening candidate generation—risk wake-up and routing enhancement—parallel category determination—judgment decision and low-confidence retry—post-verification and experience accumulation," high-quality, low-cost, and evolvable automated annotation of the data to be annotated is achieved. The overall architecture diagram of the multi-agent self-evolving data annotation system is shown below. Figure 5 As shown, the modules and their interrelationships are as follows: the orchestration and scheduling module 501 is located at the top level, driving the pre-screening candidate generation module 502, the risk wake-up and routing module 503, the category agent pool 504, and the judge decision module 505 downwards; the post-verification and active learning module 506 connects to the manual review entry point; the experience and task configuration module 507 provides labels, models, thresholds, and historical experience configurations for each agent and threshold determiner, and receives feedback from manual review. This example uses risk identification as an illustration. In other scenarios, the risk wake-up and routing module can be replaced with the wake-up and routing module of the corresponding scenario, and the category agent pool can also be replaced with the category agent pool of the corresponding scenario.

[0067] In one implementation, the system first uses the pre-screening module (PreScreener) in the pre-screening candidate generation module 502 to call the main model and multiple competing or auxiliary models in parallel. Each model returns the first-ranked (Top-1) category and probability, and the main model additionally returns the second-ranked (Top-2) category and probability. The system determines the sample certainty based on the consistency of the Top-1 identifiers (IDs) of each model and the probability difference between the Top-1 and Top-2 of the main model. When the difference is large enough, only the Top-1 candidates are retained; when the difference is small, both Top-1 and Top-2 are retained, and the Top-1 of the auxiliary models are merged to form a deduplicated candidate set. For samples that are all pre-screened as normal but may still have potential risks, the system can further call the risk agent in the risk wake-up and routing module 503 to perform a risky / no-risk binary classification. If the risk agent determines that there is risk, early exit is prohibited, and the system is forced to enter the determination stage of the router agent and the category agent. The Router Agent can supplement more categories from all categories, such as the top three (Top-3) of the entire category set, as candidates to prevent the pre-screening model from missing recalls.

[0068] For a non-empty candidate set, the system dynamically instantiates a classification agent (Category Agent) carrying exclusive knowledge and historical experience for each candidate category—that is, an expert classification agent. All Category Agents output yes / no (YES / NO) and confidence levels in parallel and independently. When no Category Agent outputs YES, the Judge Agent does not directly determine "normal," but instead receives all categories as a fallback and reselects the most suitable category. If the judge confidence level is insufficient, the system carries the complete historical record back to the pre-screening stage for retrying until the confidence threshold or the maximum number of retryes is reached. The final result can be post-verified: when the Top-1 given by the main model with high confidence is inconsistent with the final annotation, the system forces manual review. The system forms a continuous self-evolving closed loop through case accumulation, rule induction, and human feedback feedback, and adapts to different annotation tasks through a pluggable label system, model registry, Agent templates, and threshold configurations.

[0069] like Figure 5 As shown, the system framework of this disclosure embodiment includes the following modules: The orchestration and scheduling module 501 acts as the central controller of the system, responsible for the sequential scheduling and inter-stage data transfer of stages such as pre-screening, risk wake-up, routing, category determination, judge arbitration, retry, and post-verification. This module uses a semaphore mechanism to control the number of samples processed simultaneously, preventing downstream resource overload. It can interact with other modules to set the scheduling order between them. Furthermore, it can configure each module to dynamically determine the jump path based on branch signals returned from each stage (empty candidates, prohibition of early exit, single positive, all negative, insufficient confidence, inconsistent post-verification, etc.). It maintains the retry count and historical records for each sample, driving the backtracking retry logic. The input is the batch of samples to be labeled, and the output is the labeling result set and manual review tags.

[0070] The pre-screening candidate generation module 502 includes a main model adapter, an auxiliary model adapter, a consistency checker, a Top-2 threshold checker, and a candidate set generator. The main model adapter returns the category IDs, category names, and probabilities of the Top-1 and Top-2 candidates. The auxiliary model adapter calls one or more auxiliary models in parallel and returns the Top-1 results uniformly. The consistency checker checks whether the Top-1 IDs of all models are consistent; the Top-2 threshold checker calculates the probability difference between the main model's Top-1 and Top-2 candidates and determines whether Top-2 should enter the candidate set. The candidate set generator deduplicates the main model candidates, the auxiliary model's Top-1 candidates, and the "normal" category, where "normal" category IDs, such as "ID 0," can participate in subsequent competition instead of being simply discarded.

[0071] Risk Awakening and Routing Module 503: In risk identification application scenarios, it includes optional sub-components such as the Risk Agent and the Router Agent. The Risk Agent performs binary classification of samples as risky / risk-free. It can be configured to trigger only when all pre-screening models determine the samples are normal, or it can be configured to trigger in parallel with the pre-screening stage. When all pre-screening results are normal but the Risk Agent determines the samples are risky, the system prevents early exit and forces the system to proceed to the subsequent Agent process. The Router Agent selects the top three (Top-3) possible categories from the complete label system and merges them with the pre-screening candidate set to remove duplicates, supplementing the risk categories that the pre-screening models did not recall.

[0072] Category Agent Pool 504: This pool includes sub-components such as an Agent factory, a knowledge injector, a parallel executor, and a result aggregator. The Agent factory dynamically instantiates Category Agents based on the candidate category set, with each Category Agent bound to a complete knowledge context of a candidate category. The knowledge injector assembles prompt text for each Category Agent, including category definitions, decision rules, positive and negative examples, boundary cases, and historical cases and inductive rules extracted from the experience management module, rendered using a template engine. The parallel executor drives all Category Agents to execute asynchronously and in parallel, without dependency on each other; the failure of a single Category Agent does not block other Category Agents. The result aggregator collects the positive or negative (YES / NO) decision results and confidence levels of all Agents, counts the number of YES decisions, and determines the subsequent branch direction.

[0073] Among them, the risk wake-up and routing module 503 and the Category Agent pool 504 can be two independent modules, or they can be regarded as a single module, such as the risk wake-up and Category Agent pool module.

[0074] The Judge decision module 505 includes an evidence aggregation submodule, a full-category fallback submodule, a hierarchical priority submodule, a confidence calibrator, and a divergence logger. The evidence aggregation submodule presents the Category Agent's judgment results in a structured manner. When the number of YES results is 0, the full-category fallback submodule provides the Judge Agent with a complete list of categories, allowing it to select the most appropriate risk category from all categories, outputting "normal" only when it is certain the sample is normal. The hierarchical priority submodule maintains the hierarchical relationship of the label system, prioritizing finer-grained labels when conflicting labels have parent-child relationships. The confidence calibrator calibrates the final confidence level based on the consistency of the Category Agent's opinions. The divergence logger records minority opinions for the experience learning module to analyze label boundaries. Inputs include the Category Agent's judgment results, the full category list, and historical retry records; outputs include the final label, Judge confidence level, divergence information, and human review flags.

[0075] The post-validation and active learning module 506 includes sub-components such as a high-confidence validator for the main model, a hard sample identifier, a manual review queue, and a feedback loop. The high-confidence validator for the main model forces the sample to be manually reviewed when the Top-1 probability of the main model reaches a preset high-confidence threshold and is inconsistent with the final label, preventing the final result from deviating from the strong signal pre-screening conclusion. The hard sample identifier filters samples requiring manual intervention based on Judge confidence, Category Agent divergence, number of retries, and post-validation results. The manual review queue sorts hard samples by Judge confidence and validation risk, and includes a complete multi-stage judgment process record for manual reference. The feedback loop receives the manual review results, writes them as high-confidence cases into the experience management module, triggers rule updates, and forms a closed loop.

[0076] The Experience and Task Configuration Module 507 includes a case repository, a rule induction engine, an experience injector, and a task configurator. The case repository stores labeled cases in a tag-based partition, using a first-in, first-out (FIFO) strategy to control capacity limits. Manually reviewed cases are marked as high priority and are unaffected by the elimination strategy. When the number of cases for a certain tag reaches a preset threshold, the rule induction engine calls a large language model to analyze the case set and automatically generates a structured rule set containing judgment rules and boundary conditions. The task configurator maintains parameters such as the tag system, model registry, Risk / Router switch, Agent template, Top-2 threshold, Judge confidence threshold, maximum number of retries, and high-confidence verification threshold for the main model.

[0077] The data flow between the above modules is as follows: the experience and task configuration module provides labels, models, thresholds, and experience parameters to all modules; the orchestration and scheduling module sequentially drives the pre-screening candidate generation module, the risk awakening and routing module, the CategoryAgent pool, the Judge decision module, and the post-verification and active learning module; the experience and task configuration module provides experience injection to the CategoryAgent and Judge Agent, and at the same time receives the final annotation results and manual review results for case accumulation and rule summarization.

[0078] Figure 6 This is a flowchart illustrating a multi-agent self-evolutionary data annotation method 800 according to an embodiment of the present disclosure.

[0079] S610: PreScreener parallel pre-screening. For samples to be labeled. The orchestration and scheduling module calls the main model in parallel. S and K A competitor or auxiliary model {M1, M2, ..., M} KEach competitor or auxiliary model returns the Top-1 category ID, category name, and probability; the main model additionally returns the Top-2 category ID, category name, and probability. S ( x )={( s 1, n 1, p 1),( s 2, n 2, p 2)}, M k ( x )=( m k , n k , q k ) in, s 1 and s 2 represents the Top-1 and Top-2 category IDs of the main model, n 1 and n 2 represents the corresponding category name. p 1 and p 2 represents the corresponding probability; m k , n k , q k The first k The top-1 category ID, category name, and probability of each competing or auxiliary model. The competing or auxiliary models are selected as those that satisfy the unified adapter protocol.

[0080] S620: Consistency Judgment and Top-2 Threshold Determination for the Main Model. Summarize the Top-1 category IDs of the main model and all competing or auxiliary models, and calculate the consistency index: "consistency"=1[|{s_1,m_1,…,mK}|=1] Where 1[⋅] is the indicator function, which takes a value of 1 when all models have the same Top-1 ID. Further calculation of the probability difference between the main model's Top-1 and Top-2 is performed: Δ S =p1-p2 If Δ S If ≥1-τ2, where τ2 is the Top-2 threshold, it indicates that the probability of Top-2 in the main model is sufficiently low and the certainty of Top-1 determination is high, and only the Top-1 of the main model is added to the candidate set; if Δ S If <1-τ2, it indicates that the difference between Top-1 and Top-2 is small, and the candidate set is added to both the main model Top-1 and Top-2.

[0081] S630: Candidate category aggregation and competition with normal categories. The system merges and deduplicates the candidate categories of the main model obtained in step S620, the Top-1 categories of all competing or auxiliary models, and the necessary normal category ID 0, to obtain the candidate category set L. cand : L cand =dedup(L S ∪{m k |k=1,…,K}∪L0) Among them, L S The candidate set for the main model; when the normal category needs to participate in the competition for the risk category, L0={0}, otherwise L0=∅. By retaining the normal category to participate in the competition, the system can avoid prematurely excluding normal labels on samples at the boundary between "normal" and "abnormal".

[0082] S640: Risk Agent. In optional configuration, the system invokes the Risk Agent to perform binary classification on sample x, outputting risk(x) ∈ {risky, no risk}. The Risk Agent supports two triggering modes: conditional mode, triggered only when all pre-screening models classify it as normal; and parallel mode, triggered in parallel with the PreScreener. If all pre-screening models classify it as normal and the Risk Agent classifies it as risky, the system prevents early exit and forces the sample into the Router Agent and subsequent Category Agent classification process.

[0083] S650: Router Agent Candidate Supplementation and Empty Candidate Handling. In optional configuration, the Router Agent receives sample x and the current candidate set L. cand and the complete set of categories L all Select the top-3 potential categories from all categories and merge them into the candidate set: L' cand =dedup (L cand ∪RouterTop3(x,L all )) If the candidate set is not empty, then proceed to the Category Agent parallel decision-making stage.

[0084] S660: Parallel Category Agent Decision. For the candidate category set L' cand Each category in i Dynamically instantiate Category Agent A i The knowledge injector is for each A iAssembly prompt text, including categories The definitions, judgment rules, positive example set, negative example set, boundary cases, and historical cases and inductive rules extracted from the experience management module are all included. All Category Agents execute asynchronously and in parallel, each independently outputting a judgment result for sample x. A i (x)=(belongs i ,confidence i reason i ,sub_label i ) Among them, belong i ∈{True,False} is a confidence judgment. i ∈[0,1] represents the confidence level, and reason i As the basis for reasoning, sub_label i For sub-tags (if any). The aggregator then counts the number N of confirmed decisions. yes =|A i |belongs i =True}|。If N yes =1, directly adopt the result that uniquely affirms the Agent; if N yes >1, proceed to Judge Agent's comprehensive decision-making; if N yes =0, instead of directly determining it as normal, it enters the Judge Agent and provides the Judge Agent with the complete set of categories L. all As a fallback.

[0085] S670: Judge Agent's Integrated Judgment and Low-Confidence Retry. Judge Agent J receives the Category Agent's judgment result, the complete category set fallback, the original sample x, and historical retry records H. When multiple YES results exist, Judge Agent resolves conflicts by integrating the labels, confidence levels, and reasoning bases of each Category Agent; when N... yes When the confidence level is 0, the JudgeAgent determines from the complete category set whether a sample is indeed risky but not recalled by candidate categories, and only outputs "normal" when it is certain that the sample is normal content. The JudgeAgent further calibrates the confidence level based on the consensus of the Category Agents, the probability of the main model, and historical retry information. J : Judge Agent outputs the final ruling (label)J ,confidence J reason J dissent J ), where dissent J Record minority opinions for experiential learning and analysis. (If confidence...) J If the value is greater than or equal to θ, where θ is the Judge confidence threshold, then the output label is ≥ θ. J As the final annotation result; if confidence J If the number of retries is less than θ and the maximum number of retries has not been reached, then the candidate set, Category Agent result, Judge result, and low confidence reason for the current round are appended to the history H, and the process returns to step S610 to be re-executed; if the maximum number of retries has been reached, then the current best result is output and needs_review=True is marked.

[0086] Historical records can be represented as: Among them, round t For the current round number, A (t) Let t be the set of Category Agent decision results. This represents the final annotation result for round t. Let t be the confidence level in round t; The historical record H is the reason for the low confidence in round t. During retry, the historical record H is injected into the prompt text of the Category Agent, Router Agent, and Judge Agent, enabling them to analyze the reasons for the low confidence in the previous round and avoid repeating the same judgment path, thus achieving asymptotic convergence of the judgment results.

[0087] S680: Post-validation, experience accumulation, and active learning. For the final annotation results, the system performs high-confidence post-validation of the main model: if the Top-1 probability p1 ≥ θ of the main model... SS Furthermore, the top-1 category s1 of the main model and the final label J If there is a discrepancy, the requirement to review needs is enforced by setting needs_review=True, where θ SS It can be set to 0.95. The system then performs experience accumulation and active learning operations: (a) the labeled samples and their judgment results are stored in the corresponding label's experience library (or case library), and a first-in-first-out strategy is used to control the upper limit E of the case capacity for each label partition. max (a) Cases manually reviewed are marked as high priority and are not affected by the elimination strategy; (b) When the number of cases for a certain label reaches the induction threshold E triggerAt that time, the large language model is called to analyze all cases under the label, automatically summarize 3-5 judgment rules and boundary conditions, and generate a structured rule set for subsequent use by the Category Agent; (c) Identify difficult samples that meet one of the following conditions and push them to the manual review queue: Judge confidence J <θ hard (d) The results of manual review are returned to the experience base as high-confidence cases. If the induction threshold is triggered, rule induction is re-executed to cover the old rules, forming a continuous self-evolutionary closed loop of "labeling → difficult sample discovery → manual confirmation → experience accumulation → rule evolution".

[0088] The self-evolving data annotation method and system based on multi-agent cooperation proposed in this disclosure, through a task-independent pluggable configuration protocol, can be widely applied to the following data annotation scenarios: (1) Content security classification and labeling. User-generated content is classified and labeled with multiple categories, covering dozens of risk categories such as violence and illegality. The PreScreener calls the main model and competing or auxiliary models in parallel, and generates candidate categories based on Top-1 consistency and the probability difference between the main model's Top-2 and Top-2. The Risk Agent is used to discover samples that are pre-screened as normal but still have potential risks. The Router Agent supplements the candidates from the complete risk categories. The Category Agent loads exclusive judgment rules and boundary cases for each candidate risk category for in-depth analysis. In this scenario, the label hierarchy is rich, and the Judge Agent's full-category fallback and hierarchical priority mechanism can effectively handle missed calls and parent-child category conflicts.

[0089] (2) Sentiment Analysis and Labeling. Multi-level sentiment polarity labeling is applied to product reviews, social media texts, etc., such as very negative, negative, neutral, positive, very positive, etc. The primary pre-screening model returns the Top-1 / Top-2 sentiment levels, and the auxiliary model returns the Top-1 sentiment level. The system generates candidate sentiment levels based on consistency and the Top-2 gap. The Category Agent loads the corresponding judgment rules for each sentiment level. The experience-based self-evolution mechanism can continuously accumulate difficult-to-judge cases containing sarcasm and irony, and automatically summarize the corresponding judgment rules.

[0090] (3) Named Entity Recognition (NER) Labeling. Named entity recognition is performed on texts such as news articles and scientific documents to identify entities such as names of people, organizations, products, and technical terms, as well as their boundaries. The main Named Entity Recognition (NER) model returns Top-1 / Top-2 entity type candidates, while an auxiliary dictionary or lightweight model returns Top-1 candidates. Category Agents are assigned tasks according to entity type, each responsible for identifying entities of the corresponding type in the entire text. Judge Agents handle entity boundary conflicts, type ambiguities, and candidate omissions. This scenario verifies the ability of this embodiment to extend from whole-sentence classification labeling to sequence labeling paradigms.

[0091] (4) General classification labeling. Applicable to various text classification tasks such as intent recognition, topic classification, and document classification. By replacing the label system definition, model registry, Risk / Router switch, and Category / JudgeAgent template in the task configuration, the same system framework can directly adapt to different classification labeling requirements without modifying the core pipeline logic.

[0092] (5) Relation Extraction and Annotation. Semantic relationships between entity pairs in the text are annotated, such as the affiliation between a person and an organization, or the production relationship between a product and a company. The Category Agent is divided according to relationship type, loading definitions, positive and negative examples, and judgment rules for each type of relationship. The Judge Agent handles conflicting situations where the same entity pair may have multiple relationships, empty candidates, or multiple relationships coexisting. This scenario verifies the generalization ability of this embodiment to structured annotation tasks.

[0093] The self-evolving data annotation method and system based on multi-agent collaboration proposed in this disclosure can bring one or more of the following technical effects to data annotation products: (1) Significantly reduce annotation costs. Through parallel pre-screening of the main model and auxiliary models in the PreScreener stage, Top-1 consistency judgment, and Top-2 gap judgment of the main model, high-deterministic samples can avoid entering the complete multi-Agent process. Only samples that are inconsistent, have small Top-2 gaps, or are awakened by the Risk Agent are handed over to the LLM Agent for processing. In actual business scenarios, about 60%-80% of samples can be deterministically diverted in the pre-screening stage, which greatly reduces the number of calls to the large language model and the consumption of computing resources. Compared with the pure LLM annotation scheme, it can reduce the inference cost by more than 50%.

[0094] (2) Improve annotation quality and consistency. By using the multi-perspective independent judgment of the parallel Category Agent and the conflict resolution mechanism of the Judge Agent, the systematic bias of a single annotation perspective is effectively avoided. Each Category Agent carries exclusive knowledge and experience data for the corresponding category for in-depth analysis. The Judge Agent makes refined decisions based on full-category fallback, hierarchical priority and confidence calibration, thereby providing more accurate and consistent annotation results in boundary cases, candidate missed recall and multi-label semantic overlap scenarios.

[0095] (3) Achieve continuous self-evolution of the system. The two-layer experience self-evolution mechanism enables the annotation system to automatically accumulate experience from each annotation process. The case accumulation layer continuously enriches the reference case library for each label, while the rule induction layer automatically discovers the judgment rules and boundary conditions through the large language model. As the amount of annotation increases, the system's judgment capability continues to improve, the cold start time of new labels is significantly shortened, and the experience library can be reused across tasks.

[0096] (4) Reduce the burden of manual review and improve the targeting of review. The active learning closed loop accurately identifies difficult samples with low confidence, high discrepancy, retry limit, conflict between Risk Agent and pre-screening, and inconsistency between high confidence post-validation of the main model. Only samples that truly require manual intervention are pushed to the review queue, along with a complete record of the multi-stage judgment process for reviewers' reference. Compared with full sampling or random sampling, the targeting and efficiency of manual review are greatly improved, and the workload of reviewers can be reduced by more than 80%. At the same time, the feedback of review results forms positive feedback, further reducing the proportion of samples that require subsequent manual intervention.

[0097] (5) Supports rapid adaptation to new tasks. The task-independent pluggable configuration protocol enables the same system framework to quickly adapt to new annotation tasks by replacing one or more of the tag system, model registry, Risk / Router switch, Agent template, and threshold parameters. The deployment cycle of new annotation tasks is shortened from the traditional development of several weeks to configuration of several hours, and the core pipeline logic does not need to be modified, reducing system maintenance costs and error risks.

[0098] Figure 7 This is a schematic diagram of the structure of an agent-based data annotation device 700 according to an embodiment of the present disclosure. In one embodiment, the device may include: The candidate generation module 701 is used to pre-screen the candidate category set of the sample data based on the classification results of the sample data by the main model and one or more auxiliary models, wherein the candidate category set includes multiple candidate categories; The classification agent module 702 is used to classify the sample data by expert classification agents corresponding to each candidate category, and obtain their respective classification results. The decision module 703 is used to make decisions based on the classification results of the expert classification agents corresponding to each candidate category, and to obtain the labeling results of the sample data.

[0099] Figure 8 This is a schematic diagram of the structure of an agent-based data annotation device 800 according to an embodiment of this disclosure. The device may include a candidate generation module 801, a classification agent module 802, and a decision module 803. The functions and roles of each module are described in the description of the modules of the agent-based data annotation device 700 described above, and will not be repeated here. In one embodiment, the candidate generation module 801 can be used to add one or more classification results of the main model to the candidate category set based on the differences between multiple classification results of the main model; and to add one or more classification results of each of the one or more auxiliary models to the candidate category set.

[0100] In one embodiment, the candidate generation module 801 can also be used to add necessary categories corresponding to the classification scenario to the candidate category set; and to perform deduplication processing on each category in the candidate category set.

[0101] In one embodiment, the device may further include: The wake-up module 804 is used to wake up the preliminary classification agent to perform preliminary classification on the sample data and obtain the preliminary classification result. The preliminary classification result is used to indicate whether the sample data needs to be classified by the classification agent pool.

[0102] In one implementation, the triggering method for waking up the preliminary classification agent includes one or more of the following: A consistency judgment is made on the classification results of the main model and the one or more auxiliary models to obtain a consistency result; It is triggered in parallel with the function of filtering the set of candidate categories.

[0103] In one embodiment, the device may further include: The routing module 805 is used to receive the sample data, the candidate category set, and the complete category set through the routing agent; and to select a portion of the categories from the complete category set to add to the candidate category set.

[0104] In one embodiment, the device may further include: The classification agent pool 806 is used to instantiate corresponding expert classification agents for each candidate category in the candidate category set; to assemble prompt text for the expert classification agents corresponding to each candidate category, wherein the prompt text includes one or more of the meaning of its own candidate category, judgment rules, positive example set, negative example set, boundary cases, historical cases, and induction rules; and to asynchronously and in parallel classify the sample data based on the prompt text through the expert classification agents corresponding to each candidate category to obtain their respective classification results.

[0105] In one implementation, the decision module 803 is further configured to perform one or more of the following steps: When only one expert classification agent outputs a positive result, the labeling result classification of the sample data is obtained according to the candidate category corresponding to the expert classification agent that outputs the positive result; If no expert classification agent in the candidate category set outputs a positive result, a complete category set is provided to the judgment agent, which is used to obtain the labeling result of the sample data based on the complete category set. When multiple expert classification agents output positive results, the judgment agent makes a decision by combining the classification results and reasoning basis of the expert classification agents corresponding to each candidate category, and obtains the labeling result of the sample data.

[0106] In one implementation, the decision module 803 makes a decision by comprehensively outputting the classification results and reasoning basis of the expert classification agents corresponding to each candidate category through the judgment agent, and obtains the labeling results of the sample data, including: The confidence level is calibrated based on one or more of the following: the degree of consensus among the expert classification agents corresponding to each candidate category, the probability in the classification results of the main model, and historical retry information. The labeling result of the sample data is determined based on the confidence level of the judgment.

[0107] In one implementation, the decision module 803 determines the labeling result of the sample data based on the confidence level, including one or more of the following steps: If the confidence level is less than the confidence threshold, the candidate category set for the current round, the classification result of the expert classification agent, the judgment result, and the reason for the low confidence level are added to the history. If the maximum number of retries has not been reached, the agent-based sample data annotation device will be executed again. If the maximum number of retries is reached, output the current best result and add a mark indicating that the sample data needs to be reviewed.

[0108] In one embodiment, the device may further include: The post-verification module 807 is used to add a verification mark to the sample data when the probability of the first classification result of the main model is greater than the confidence threshold, but the first classification result is inconsistent with the annotation result.

[0109] In one embodiment, the device may further include: Case Library 808 is used to store sample data with labels that require verification and their annotation results.

[0110] In one embodiment, the device may further include: The review push module 809 is used to push sample data that meets the difficulty criteria from the sample data that has the mark of needing review to the manual review queue.

[0111] In one implementation, the difficulty conditions include one or more of the following: the confidence level is less than the difficulty threshold; the ratio of positive to negative results of the expert classification agent is less than the proportion threshold; the number of retries reaches the upper limit; the classification result of the preliminary classification agent conflicts with the pre-screening conclusion; and the post-validation of the main model fails.

[0112] In one implementation, the case library 808 is also used to store the received manual review results.

[0113] In one embodiment, the device may further include: The rule summarization module 810 is used to summarize the cases in the case library according to the specified categories to obtain judgment rules and boundary conditions, so as to generate a structured rule set for the expert classification agent corresponding to the specified category.

[0114] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0115] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0116] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0117] Figure 9A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0118] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0119] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0120] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as agent-based data annotation methods. For example, in some embodiments, the agent-based data annotation method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the agent-based data annotation method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform agent-based data annotation methods by any other suitable means (e.g., by means of firmware).

[0121] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0123] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0125] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0126] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0127] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0128] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data annotation method based on intelligent agents, comprising: Based on the classification results of the sample data by the main model and one or more auxiliary models, a candidate category set of the sample data is pre-screened, and the candidate category set includes multiple candidate categories; The sample data is classified by an expert classification agent corresponding to each candidate category, and the classification results are obtained respectively. The labeling results of the sample data are obtained by making decisions based on the classification results of the expert classification agents corresponding to each candidate category.

2. The method according to claim 1, wherein, Based on the classification results of the sample data by the main model and one or more auxiliary models, a candidate category set for the sample data is pre-screened, including: Based on the differences between multiple classification results of the main model, one or more classification results of the main model are added to the candidate category set; Add one or more classification results from each of the one or more auxiliary models to the candidate category set.

3. The method according to claim 2, wherein, Based on the classification results of the sample data by the main model and one or more auxiliary models, the candidate category set of the sample data is pre-screened, and the method further includes: Add the necessary categories corresponding to the classification scenario to the candidate category set; The candidate category set is deduplicated.

4. The method according to any one of claims 1 to 3, further comprising: The preliminary classification agent is awakened to perform a preliminary classification of the sample data, and the preliminary classification result is obtained. The preliminary classification result is used to indicate whether the sample data needs to be classified by the classification agent pool. The triggering methods for waking up the preliminary classification agent include one or more of the following: A consistency judgment is made on the classification results of the main model and the one or more auxiliary models to obtain a consistency result; It is triggered in parallel with the function of filtering the set of candidate categories.

5. The method according to any one of claims 1 to 4, further comprising: The sample data, the candidate category set, and the complete category set are received by the routing agent; Select a subset of categories from the complete category set and add them to the candidate category set.

6. The method according to any one of claims 1 to 5, wherein, The sample data is classified using expert classification agents corresponding to each candidate category, resulting in classification results for each category, including: Based on the classification agent pool, an expert classification agent is instantiated for each candidate category in the candidate category set; Assemble prompt text for the expert classification agent corresponding to each candidate category. The prompt text includes one or more of the following: the meaning of its own candidate category, the judgment rule, the set of positive examples, the set of negative examples, the boundary cases, the historical cases, and the induction rule. The expert classification agents corresponding to each candidate category asynchronously and in parallel classify the sample data based on the prompt text to obtain their respective classification results.

7. The method according to any one of claims 1 to 6, wherein, The process involves making decisions based on the classification results of expert classification agents for each candidate category, and obtaining the annotation results of the sample data, including one or more of the following steps: When only one expert classification agent outputs a positive result, the labeling result classification of the sample data is obtained according to the candidate category corresponding to the expert classification agent that outputs the positive result; If no expert classification agent in the candidate category set outputs a positive result, a complete category set is provided to the judgment agent, which is used to obtain the labeling result of the sample data based on the complete category set. When multiple expert classification agents output positive results, the judgment agent makes a decision by combining the classification results and reasoning basis of the expert classification agents corresponding to each candidate category, and obtains the labeling result of the sample data.

8. The method according to claim 7, wherein, The decision-making agent comprehensively outputs the classification results and reasoning basis of the expert classification agents corresponding to each candidate category to make a decision, thereby obtaining the labeled results of the sample data, including: The confidence level is calibrated based on one or more of the following: the degree of consensus among the expert classification agents corresponding to each candidate category, the probability in the classification results of the main model, and historical retry information. The labeling result of the sample data is determined based on the confidence level of the judgment.

9. The method according to claim 8, wherein determining the labeling result of the sample data based on the confidence level comprises one or more of the following steps: If the confidence level is less than the confidence threshold, the candidate category set for the current round, the classification result of the expert classification agent, the judgment result of the decision agent, and the reason for the low confidence level are added to the history record. If the maximum number of retries has not been reached, the agent-based sample data annotation method will be re-executed; If the maximum number of retries is reached, output the current best result and add a mark indicating that the sample data needs to be reviewed.

10. The method according to any one of claims 1 to 9, further comprising one or more of the following steps: If the probability of the first classification result of the main model is greater than the confidence threshold, but the first classification result is inconsistent with the labeled result, a verification mark is added to the sample data. Add the sample data that requires verification and its annotation results to the case library; Sample data that meets the difficulty criteria from the sample data that needs to be reviewed are pushed to the manual review queue; wherein, the difficulty criteria include one or more of the following: the confidence level is less than the difficulty threshold; the ratio of positive results to negative results of the expert classification agent is less than the proportion threshold; the number of retries reaches the upper limit; the classification result of the preliminary classification agent conflicts with the pre-screening conclusion; the post-validation of the main model fails. The received manual review results will be added to the case library.

11. The method of claim 10, further comprising: The cases in the case library are categorized according to a specified category to obtain judgment rules and boundary conditions, thereby generating a set of structured rules corresponding to the specified category. The set of structured rules corresponding to the specified category is used to instantiate the expert classification agent corresponding to the specified category.

12. A data annotation device based on intelligent agents, comprising: The candidate generation module is used to pre-screen the candidate category set of the sample data based on the classification results of the main model and one or more auxiliary models. The candidate category set includes multiple candidate categories. The classification agent module is used to classify the sample data by using expert classification agents corresponding to each candidate category, and obtain their respective classification results; The decision module is used to make decisions based on the classification results of the expert classification agents corresponding to each candidate category, and to obtain the labeling results of the sample data.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images