Data annotation method and device, equipment and storage medium

Abstract

The invention discloses a data annotation method and device, equipment and a storage medium. The method comprises the following steps: firstly, constructing an initial mode pool comprising a plurality of different source large models, and deploying the initial mode pool on line to serve real users; and in response to to-be-labeled data uploaded by a real user, combining the to-be-labeled data with a preset cue word to generate a labeled cue word, and inputting the cue word into each large model in the mode pool to obtain a plurality of predicted labels of the to-be-labeled data. And determining at least one output label from the plurality of prediction labels based on a multi-model voting consensus decision-making mechanism, outputting the output label to a real user to obtain user feedback of the real user for the output label, and generating a training sample in combination with the user feedback and the to-be-labeled data. And repeatedly executing the steps of responding to the to-be-labeled data, performing statistical voting and obtaining user feedback until a preset period is reached, and stopping to obtain a plurality of training samples. According to the method and the device, the labeling efficiency and accuracy are improved, the manual dependence is reduced, and meanwhile, the quality of the labeled data is ensured.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a data annotation method, apparatus, device, and storage medium. Background Technology

[0002] Currently, in task processing within specific professional fields such as medicine, finance, and law, mainstream optimization solutions primarily rely on supervised fine-tuning of large-scale foundational models as their core technical approach. This involves updating the parameters of pre-trained models using domain-specific, high-quality training data to adapt them to the specific needs of the scenario. However, this approach faces a critical bottleneck: the annotation of high-quality training data still largely depends on manual work. This not only results in low annotation efficiency, making it difficult to quickly meet the large-scale data requirements of model training, but also carries the risk of errors in the annotation results due to subjective judgment biases and operational oversights, thereby affecting the effectiveness and reliability of subsequent model fine-tuning.

[0003] Therefore, how to improve the efficiency and accuracy of training data annotation, reduce reliance on manual annotation, and at the same time ensure the quality of the annotated data is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a data annotation method, apparatus, device, and storage medium that can improve the efficiency and accuracy of training data annotation, reduce reliance on manual annotation, and ensure the quality of the annotated data.

[0005] The embodiments of this application disclose the following technical solutions:

[0006] A data annotation method, the method comprising:

[0007] An initial model pool is constructed and deployed to serve real users; the initial model pool includes multiple large models from different sources.

[0008] In response to the data to be labeled uploaded by the real user, the annotation prompt words are generated by combining the data to be labeled and the preset prompt words. The annotation prompt words are then input into the various large models in the initial model pool to obtain multiple predicted annotations for the data to be labeled.

[0009] Based on a multi-model voting consensus decision-making mechanism, at least one output label is determined from the multiple prediction labels and output to the real user.

[0010] Obtain user feedback from real users regarding the output annotations, and generate training samples by combining the user feedback and the data to be annotated; the user feedback includes an adoption status and a correction operation; the adoption status is used to indicate whether the output annotation has been adopted; the correction operation includes the adjustment content of the output annotation given by the real users, wherein the correction operation is empty when the adoption status indicates that the output annotation has been adopted;

[0011] Repeat the steps of responding to the labeled data, counting votes, and obtaining user feedback until the preset period is reached, then stop to obtain multiple training samples.

[0012] In one possible implementation, generating training samples by combining the user feedback and the data to be labeled includes:

[0013] If the user feedback indicates that the output label is adopted, then the output label is applied to the data to be labeled to obtain the training sample;

[0014] If the user feedback indicates that the output label is not adopted, the output label is adjusted based on the correction operation to obtain an adjusted label, and the adjusted label is labeled on the data to be labeled to obtain the training sample.

[0015] In one possible implementation, the multi-model voting consensus decision-making mechanism, which determines at least one output label from the multiple prediction labels and outputs it to the real user, includes:

[0016] The large models in the initial model pool are obtained to vote on each prediction label based on the multi-model voting consensus decision mechanism;

[0017] Predicted labels with a number of votes greater than or equal to the consensus threshold are identified as output labels, and these output labels are output to the real users.

[0018] In one possible implementation, the method further includes:

[0019] The multiple training samples are divided into a test set and a training set;

[0020] A baseline model is determined, and the baseline model is then subjected to supervised fine-tuning and reinforcement learning training using the training set to obtain an optimized model.

[0021] The optimized model is tested using the test set to obtain the performance indicators of the optimized model.

[0022] When the performance indicators of the optimization model meet the preset standards, the optimization model is determined to be a usable model;

[0023] The available models are used to replace the initial model pool and deployed online to serve the real users.

[0024] In one possible implementation, the baseline model is selected from the initial model pool or external candidate models.

[0025] A data annotation device, the device comprising:

[0026] A deployment unit is built to construct an initial model pool and deploy the initial model pool to serve real users; the initial model pool includes multiple large models from different sources.

[0027] The annotation unit is used to respond to the data to be annotated uploaded by the real user, combine the data to be annotated with preset prompt words to generate annotation prompt words, and input the annotation prompt words into each large model in the initial mode pool to obtain multiple predicted annotations for the data to be annotated.

[0028] A voting output unit is used to determine at least one output label from the multiple prediction labels and output it to the real user based on a multi-model voting consensus decision-making mechanism.

[0029] The acquisition unit is used to acquire user feedback from the real user regarding the output label; the user feedback includes an adoption status and a correction operation; the adoption status is used to indicate whether the output label has been adopted; the correction operation includes the adjustment content of the output label given by the real user, wherein the correction operation is empty when the adoption status indicates that the output label has been adopted;

[0030] A sample generation unit is used to generate training samples by combining the user feedback and the data to be labeled.

[0031] The execution unit is used to repeatedly execute the steps of responding to the data to be labeled, counting votes, and obtaining user feedback until a preset period is reached, at which point it stops and obtains multiple training samples.

[0032] In one possible implementation, the sample generation unit is specifically used for:

[0033] If the user feedback indicates that the output label is adopted, then the output label is applied to the data to be labeled to obtain the training sample;

[0034] If the user feedback indicates that the output label is not adopted, the output label is adjusted based on the correction operation to obtain an adjusted label, and the adjusted label is labeled on the data to be labeled to obtain the training sample.

[0035] In one possible implementation, the voting output unit is specifically used for:

[0036] The large models in the initial model pool are obtained to vote on each prediction label based on the multi-model voting consensus decision mechanism;

[0037] Predicted labels with a number of votes greater than or equal to the consensus threshold are identified as output labels, and these output labels are output to the real users.

[0038] A data annotation device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the data annotation method described above.

[0039] A computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform the data annotation method described above.

[0040] Compared with the prior art, this application has the following beneficial effects:

[0041] This application provides a data annotation method, apparatus, device, and storage medium. Specifically, when executing the data annotation method provided in this application, an initial model pool containing multiple large models from different sources can first be constructed and deployed online to serve real users. These large models can come from different training data and architectures, exhibiting diversity and complementarity. When a real user uploads data to be annotated, annotation prompts are generated by combining the data to be annotated with preset prompts. These prompts are used to guide the various models in making annotation predictions. The generated annotation prompts are input into the various large models in the initial model pool, and multiple large models predict and annotate the same data to be annotated, thereby obtaining multiple prediction annotation results. Next, a multi-model voting consensus decision-making mechanism is adopted to determine at least one output annotation from the multiple prediction annotations and output it to the real user. This mechanism can improve the accuracy and reliability of the annotation results. Then, user feedback from real users regarding the output annotation is obtained, including adoption status and correction operations. The adoption status indicates whether the output annotation has been adopted, and the correction operation includes the user's adjustments to the annotation result. If the output annotation has been adopted, the correction operation is empty. High-quality training samples are generated by combining user feedback and the data to be annotated. Finally, repeat the steps of responding to the labeled data, counting votes, and obtaining user feedback until the preset cycle is reached. Through continuous iteration, a large number of training samples can be generated, which can be used to further optimize the large model and improve the quality and efficiency of subsequent annotation results.

[0042] This application constructs an initial model pool of large models from multiple different sources and leverages a multi-model voting consensus decision-making mechanism to quickly output highly reliable annotation results without relying on pre-annotated training data. This significantly reduces reliance on manual annotation and improves the accuracy of initial annotations through cross-validation between models. Furthermore, this application deeply couples user feedback with the data to be annotated to generate training samples, transforming explicit corrections and implicit adoption behaviors from real-world scenarios into structured labeled data. This not only synchronizes the annotation process with actual application scenarios, significantly improving data annotation efficiency, but also continuously optimizes annotation quality based on accumulated real-world feedback. It perfectly solves the pain points of traditional manual annotation, such as low efficiency, difficulty in controlling errors, and limited data scale, achieving a synergistic improvement in annotation efficiency, accuracy, and data quality. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A flowchart illustrating a data annotation method provided in this application embodiment;

[0045] Figure 2 A flowchart of a model optimization method provided in this application embodiment;

[0046] Figure 3 This is a schematic diagram of the structure of a data annotation device provided in an embodiment of this application. Detailed Implementation

[0047] To facilitate understanding of the technical solutions provided in the embodiments of this application, the background technology involved in the embodiments of this application will be described below.

[0048] Currently, for specific professional domain tasks, mainstream solutions mainly rely on supervised fine-tuning of a large base model. This fine-tuning method requires high-quality training data, which is currently usually obtained through manual annotation. However, manual annotation is not only inefficient, but also susceptible to human error throughout the annotation process, thus affecting data quality and the performance of the final model.

[0049] To address this issue, this application provides a data annotation method, apparatus, device, and storage medium. First, an initial model pool is constructed and deployed as an online service for real users. This initial model pool contains multiple large models from different sources, capable of handling diverse annotation tasks. When a real user uploads data to be annotated, corresponding annotation prompts are generated by combining this data with preset prompts. These prompts are then input into various large models in the initial model pool to obtain multiple predicted annotations. Next, a multi-model voting consensus decision mechanism selects one or more optimal output annotations from these predicted annotations and provides them to the user. Furthermore, user feedback on the output annotations is obtained. This feedback includes not only whether the user has adopted the annotation but also possible adjustments to the annotation content. When the user adopts the annotation, the correction operation is empty; if not adopted, the user can provide specific corrections. This process ensures deep coupling between user feedback and the data to be annotated, generating high-quality training samples. Finally, the entire process is repeated until a preset period is reached, thereby continuously accumulating training samples. This application constructs an initial model pool containing multiple large models from different sources. Utilizing a multi-model voting consensus decision-making mechanism, it can quickly output highly reliable annotation results without relying on pre-annotated training data, thus significantly reducing reliance on manual annotation. Furthermore, cross-validation between models improves the accuracy of the initial annotations. Simultaneously, this application deeply couples user feedback with the data to be annotated to generate new training samples, transforming explicit corrections and implicit adoption behaviors from real-world scenarios into structured labeled data. This design not only synchronizes the annotation process with actual application scenarios, significantly improving data annotation efficiency, but also continuously optimizes annotation quality based on accumulated real-world feedback. It fundamentally solves the problems of low efficiency, uncontrollable errors, and limited data scale inherent in traditional manual annotation, achieving a synergistic improvement in annotation efficiency, accuracy, and data quality.

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0051] See Figure 1 The figure is a flowchart of a data annotation method provided in an embodiment of this application, as shown below. Figure 1 As shown, the data annotation method may include steps S101-S105:

[0052] S101: Construct an initial pattern pool and deploy the initial pattern pool to serve real users.

[0053] To overcome the limitations of single models in terms of insufficient knowledge coverage and limited decision-making reliability in specialized fields, and to rapidly build an initial service architecture that meets high accuracy requirements, this application first constructs an initial model pool containing multiple large models from different sources. The core idea is to leverage models with different technical backgrounds, training data, and architectural designs to form a heterogeneous complementarity, thereby enabling prediction of specialized tasks from multiple perspectives and avoiding the cognitive blind spots of a single model. After completing the model pool, it is fully deployed to the production environment and officially launched, directly providing prediction and annotation services to real users. This not only provides practical application scenario support for subsequent multi-model collaborative decision-making but also accumulates real business data and user feedback for the subsequent launch of a data self-circulation framework.

[0054] S102: In response to the data to be labeled uploaded by the real user, generate labeling prompts by combining the data to be labeled and preset prompts, and input the labeling prompts into each of the large models in the initial model pool to obtain multiple predicted labels for the data to be labeled.

[0055] To ensure that the various heterogeneous large models in the initial model pool accurately meet the annotation needs of professional fields, when the system receives data to be annotated uploaded by real users, it will first combine the attributes of the data itself with the preset professional field prompts to generate highly targeted annotation prompts. This not only clarifies the core requirements of the annotation task (such as label category and annotation rules) but also fits the scene characteristics of the data to be annotated, avoiding prediction bias caused by general prompts.

[0056] For example, taking the insurance claim case type classification task as an example, if the user uploads an insurance report involving a vehicle collision, the preset prompt is "Please label the following insurance report based on the four categories of tags: 'single-vehicle accident,' 'two-vehicle accident,' 'multi-vehicle accident,' and 'false report,' and only output the matching tag names." The system will combine the preset prompt with the key information of this vehicle collision report (such as accident description, accident location, and statements from the parties involved) to generate a unique labeling prompt: "Please label the following insurance report (text content: The insured drove the vehicle to the XX section of the road on XX date, and crashed into the guardrail due to improper operation to avoid a pedestrian. The front of the vehicle was damaged. There were no other parties involved...) based on the four categories of tags: 'single-vehicle accident,' 'two-vehicle accident,' 'multi-vehicle accident,' and 'false report,' and only output the matching tag names." Subsequently, the customized annotation prompts are input into four different source models (A, B, C, and D) in the initial model pool. Each model will independently complete the prediction based on its own training knowledge and reasoning logic in the insurance field, and finally output four sets of prediction annotations corresponding to the reported data (for example, model A outputs "single-party accident", model B outputs "single-party accident", model C outputs "double-party accident", and model D outputs "single-party accident"), providing a rich heterogeneous prediction foundation for subsequent multi-model voting consensus decision-making.

[0057] S103: Based on the multi-model voting consensus decision-making mechanism, at least one output label is determined from the multiple prediction labels and output to the real user.

[0058] To address the issue of insufficient prediction accuracy of single models in specialized fields, this application employs a multi-model voting consensus decision-making mechanism to aggregate and verify the prediction results of various heterogeneous models. Ultimately, high-credibility outputs are selected and labeled for feedback to real users. The core logic of this mechanism is to leverage the differences in knowledge reserves and reasoning logic among different source models to achieve cross-validation. Through "consensus filtering," misjudgments from single models are filtered out, ensuring that the output results meet the requirements of low misjudgment risk in specialized scenarios.

[0059] The specific process is as follows: First, summarize the prediction labels of all large models in the initial model pool for the same data to be labeled, and count the number of model support votes for each label (i.e. how many models output the label); then, based on the accuracy requirements of the professional field, preset a consensus threshold (this threshold is determined based on the number of large models from different sources in the initial model pool), and only the labels with a number of votes greater than or equal to the threshold are determined as the final output labels, which ensures the reliability of the results and balances the accuracy and coverage of the results through threshold control.

[0060] For example, taking the risk classification task of insurance claims as an example, the initial model pool contains four different source models A, B, C, and D, with a preset consensus threshold k=3. If a user uploads a vehicle accident claim report, after independent prediction by the four models, the "normal claim" label receives 1 vote, the "suspected insurance fraud" label receives 4 votes, the "incomplete materials" label receives 2 votes, and the "not covered by insurance" label receives 2 votes. Then, according to the consensus rules, only the "suspected insurance fraud" label, which receives ≥3 votes, is selected as the final output label and fed back to the user. This mechanism does not rely on any training data; it can improve the system's average accuracy to a high level required for professional scenarios simply through multi-model collaborative verification. It effectively avoids the cognitive blind spots and decision-making biases that may occur with a single model, perfectly adapting to the core needs of high-risk fields such as financial risk control and insurance claims.

[0061] S104: Obtain user feedback from the real users regarding the output annotations, and generate training samples by combining the user feedback and the data to be annotated.

[0062] To overcome the scarcity of labeled data in specialized fields and enable the system to continuously learn from real-world applications, this application simultaneously collects complete user feedback on the labeled data after outputting the results to users. This feedback is then deeply integrated with the original unlabeled data to generate high-quality structured training samples. The core of the user feedback includes two key pieces of information: first, the adoption status, which clearly indicates whether the user approves of the system's labeled results; and second, the correction operation, specifically referring to adjustments made by the user to unapproved labeled results (such as deleting incorrect labels or adding missing labels). If the adoption status is "adopted," the corresponding correction operation will be null, requiring no further adjustment.

[0063] For example, taking the task of labeling insurance claim cases as an example, assuming that the labeling result output by the system to the user is "single-vehicle accident", if the user approves the label and directly adopts it (adoption status is "adopted", correction operation is empty), the system will automatically combine "vehicle collision report data to be labeled + 'single-vehicle accident' label" into a valid training sample; if the user finds that the system has omitted the label related to "vehicle wading" (adoption status is "not adopted") and adds the label of "water wading insurance related accident" through operation (correction operation is "add 'water wading insurance related accident' label"), the system will take the user's corrected label as the standard and integrate "vehicle collision report data to be labeled + 'single-vehicle accident', 'water wading insurance related accident' label" into a training sample; if the user determines that the system's label of "single-vehicle accident" is incorrect and should actually be "two-vehicle accident" (adoption status is "not adopted", correction operation is "delete 'single-vehicle accident' label, add 'two-vehicle accident' label"), the system will generate the corresponding training sample according to the corrected content. In this way, actual user feedback is directly transformed into high-quality data needed for model optimization, which not only solves the problems of high cost and low efficiency of manual annotation, but also ensures a high degree of fit between training samples and real business scenarios.

[0064] S105: Repeat steps S102-S104 until the preset period is reached, then stop to obtain multiple training samples.

[0065] To continuously accumulate sufficient high-quality training samples that closely match real-world business scenarios, this solution sets the entire process from steps S102 to S104 as a periodic loop execution mode, rather than a single run. The core logic is to rely on continuous service in the production environment to allow data accumulation to progress in sync with user usage, gradually building a large-scale domain-specific dataset. Specifically, the system will repeatedly execute the entire chain of operations according to a preset fixed cycle (e.g., once a month): "S102 receives user-uploaded data to be labeled, processes it, generates customized labeling prompts, and obtains multi-model prediction labels; S103 determines output labels through a voting consensus mechanism; S104 collects user feedback and generates structured training samples." After accumulating a preset number of cycles (e.g., 6 consecutive cycles), the loop stops. At this point, through multiple rounds of real-world scenario verification and data accumulation, a batch of high-quality, highly adaptable training samples has been obtained, providing sufficient and high-quality data support for subsequent supervised fine-tuning of the benchmark model and reinforcement learning training.

[0066] It should also be noted that within each period, the system automatically cleans and denoises the collected user feedback data, filtering out invalid feedback and abnormal data to ensure the reliability of the training samples.

[0067] Based on the content of S101-S105, an initial model pool containing multiple large models from different sources is first constructed and deployed online to serve real users. When data to be labeled is received from real users, targeted labeling prompts are generated by combining the data to be labeled with preset prompts. These prompts are then input into the various large models in the initial model pool, and each model independently performs predictions to obtain multiple predicted labels corresponding to the data to be labeled. Next, a multi-model voting consensus decision-making mechanism is used to filter the multiple predicted labels, determining at least one output label to be output to the real user. User feedback on the output label is collected, including acceptance status and correction operations. The acceptance status clarifies whether the output label has been accepted by the user, and the correction operation records the adjustments made to the output label by the user (if the acceptance status is "accepted," the correction operation is empty). Then, based on the user feedback and the corresponding data to be labeled, structured training samples are generated. Finally, the entire process of "receiving data to be labeled, generating labeling prompts and obtaining multi-model predicted labels, determining output labels through voting consensus, collecting user feedback and generating training samples" is repeated until a preset period is reached, at which point it stops, ultimately yielding a batch of high-quality training samples. This application achieves a synergistic improvement in labeling efficiency, accuracy, and data quality.

[0068] In one possible implementation, the process of generating training samples by combining user feedback and the data to be labeled follows a labeling logic that is "centered on real user needs," ensuring that the training samples are highly consistent with actual business scenarios. Specifically, this includes the following two scenarios:

[0069] If the user's feedback on the adoption status clearly indicates that the output label has been adopted, it means that the labeling result generated by the system through the multi-model voting consensus decision-making mechanism meets the user's judgment needs for the professional task. At this time, the output label is directly bound to the corresponding data to be labeled, and training samples are formed in the structured form of "data to be labeled - output label". The effective labeling relationship in the real scenario can be preserved without additional adjustments.

[0070] If the user's feedback indicating the adoption status of the output label is not adopted, and includes specific correction actions (such as deleting incorrect labels, adding missing labels, replacing incompatible labels, etc.), then the user's correction actions will be used as the basis for labeling. The original output labels will be adjusted accordingly. For example, incorrect labels identified by the user may be deleted, necessary labels added by the user may be added, or labels confirmed by the user may be replaced with accurate labels, thus obtaining adjusted labels that meet the actual needs. These adjusted labels will then be bound to the data to be labeled, forming a structured training sample of "data to be labeled - adjusted labels," ensuring the accuracy and practicality of the sample labeling.

[0071] This generation method not only makes full use of the effective results of the system's initial predictions, but also corrects the deviations through user feedback, so that the training samples can truly reflect the actual annotation needs of the professional field, providing high-quality data support for subsequent model optimization, which perfectly fits the core design concept of "self-circulation of production environment data" of this invention.

[0072] In one possible implementation, the multi-model voting consensus decision-making mechanism, which determines at least one output label from the multiple prediction labels and outputs it to the real user, includes:

[0073] The large models in the initial model pool are obtained, and their votes for each prediction label are determined based on the multi-model voting consensus decision mechanism. Then, the prediction labels with a vote count greater than or equal to the consensus threshold are identified as output labels, and the output labels are output to the real users.

[0074] Specifically, the voting results are first summarized: collect the independent prediction labels of all different source large models in the initial model pool for the same data to be labeled, and calculate the number of model support votes obtained for each prediction label as a statistical unit, that is, how many large models output the label, to form a complete "label-vote correspondence".

[0075] Then, consensus threshold screening and result output are performed: First, a consensus threshold is set based on the total number of large models in the initial model pool (for example, if the model pool contains 4 large models, the threshold can be set to 3, meaning that more than half of the models must support it). The core function of this threshold is to balance accuracy and result coverage, ensuring that only highly reliable labels recognized by the majority of models are selected. Subsequently, the number of votes for all predicted labels is verified, and only labels with a number of votes greater than or equal to the consensus threshold are determined as the final output labels and fed back to the real users who uploaded the data to be labeled.

[0076] This implementation method does not rely on any training data. Through the dual verification of "multi-model voting + quantitative threshold screening", it leverages the differences in knowledge coverage and reasoning logic of different source models to form a complementary relationship, effectively filtering out misclassifications of single models and quickly achieving the high accuracy requirements of professional fields. At the same time, the binding design of threshold and model number makes the decision rules more adaptable and interpretable.

[0077] In one possible implementation, such as Figure 2 As shown, this application also includes a model optimization method, which can be implemented through steps S201-S205:

[0078] S201: Divide the multiple training samples into a test set and a training set.

[0079] To ensure the effectiveness of subsequent model training and the objectivity of performance evaluation, avoid overfitting of training data leading to insufficient generalization ability, and provide an independent and reliable basis for verifying the performance of the optimized model, it is necessary to scientifically divide the multiple structured training samples accumulated through multiple cycles of user feedback into independent test and training sets. The training set will be used for supervised fine-tuning and reinforcement learning training of the baseline model, injecting domain knowledge and annotation patterns into the model. The test set will be specifically used to evaluate the generalization performance of the optimized model, ensuring that the evaluation results are not influenced by the training data and can truly reflect the model's performance on unseen new data, providing objective and reliable decision support for whether to trigger a system architecture switch. For example, a 7:3 ratio can be used, with 70% of the samples as the training set for model learning and 30% as the test set to verify model performance.

[0080] S202: Determine the baseline model, and use the training set to perform supervised fine-tuning and reinforcement learning training on the baseline model to obtain the optimized model.

[0081] To achieve targeted upgrades and professional adaptation of model capabilities, we can first select models with better performance and stronger domain adaptability from the initial model pool, or choose external candidate models outside the initial model pool based on actual needs, and determine them as the benchmark models for subsequent optimization. This selection ensures that the benchmark models have solid basic performance and provides a clear iterative starting point for model optimization. After determining the benchmark model, we use the high-quality training set accumulated through the production environment data self-circulation framework as the core support. We first update the parameters of the benchmark model through supervised fine-tuning, enabling it to quickly learn the annotation rules, knowledge points, and task characteristics of the professional domain, thus consolidating the model's domain adaptability. On this basis, we further carry out reinforcement learning training, improving the reliability of its judgments and the stability of its results in complex scenarios by optimizing the model's reasoning logic and decision consistency. After two rounds of progressive training, we finally form an optimized model that combines professional accuracy and reasoning robustness.

[0082] In one possible implementation, the baseline model is selected from the initial model pool or external candidate models.

[0083] S203: Use the test set to test the optimized model and obtain the performance index of the optimized model.

[0084] To objectively and accurately evaluate the practical application capabilities and generalization performance of the optimized model, and to avoid overfitting the training data, comprehensive model testing can be conducted using a test set. During testing, all labeled samples from the test set that were not used in model training are input into the optimized model, allowing it to independently complete prediction tasks for these unfamiliar samples. Subsequently, professional evaluation tools are used to compare the model's output results with the true labels of the test set samples, focusing on calculating core performance metrics such as precision and recall. Precision, as a key evaluation dimension in high-risk professional domains, directly reflects the reliability of the model's prediction results, while recall ensures the completeness of task coverage. Through multi-dimensional metric calculation, comprehensive evaluation data reflecting the overall performance of the optimized model is ultimately generated, providing an objective and reliable basis for subsequent decision-making regarding whether to initiate a system architecture switch.

[0085] S204: When the performance indicators of the optimization model meet the preset standards, the optimization model is determined to be a usable model.

[0086] To ensure that the optimized model effectively meets the business needs of the professional domain and achieves an optimal balance between system performance and operating costs, clear and quantifiable pre-defined performance standards must be established for the optimized model. These standards are based on the core performance indicators of the initial multi-model integration system (e.g., precision must be consistently higher than the initial system, and auxiliary indicators such as recall must not be lower than the minimum requirements of the business scenario), while also aligning with the professional domain's core demand for low false positive risk (e.g., precision must reach above 0.9 in high-risk scenarios). After completing model testing and obtaining key performance indicators such as precision and recall, a comprehensive comparison of all indicators of the optimized model with the pre-defined standards is conducted. If all indicators meet or exceed the standard requirements, it indicates that the optimized model has the ability to independently support business services and has surpassed the initial architecture in performance. At this point, it is officially designated as a usable model, laying the foundation for subsequent system architecture switching and replacement of the initial model pool.

[0087] S205: Replace the initial model pool with the available models and deploy them online to serve the real users.

[0088] After the performance verification of the optimized model is completed and it is determined to be a usable model, the iterative upgrade process of the system architecture will be initiated. With this high-performance usable model as the core, the deployment, configuration and service switching of the production environment will be completed, and the original multi-model integration architecture (i.e., the initial model pool) will be completely replaced.

[0089] This deployment approach ensures that the service accuracy and reliability are no less than those of the initial integrated system. By simplifying the architecture and reducing the number of model calls, it significantly reduces computing power consumption and response latency. While improving service efficiency and optimizing user experience, it also effectively controls the long-term operating costs of the system, achieving the core goal of "simplified architecture, optimal efficiency and cost after performance targets are met".

[0090] See Figure 3 , Figure 3 This is a schematic diagram of a data annotation device provided in an embodiment of this application. Figure 3 As shown, the data annotation device includes:

[0091] A deployment unit 301 is constructed to build an initial model pool and deploy the initial model pool to serve real users; the initial model pool includes multiple large models from different sources.

[0092] The annotation unit 302 is used to respond to the data to be annotated uploaded by the real user, combine the data to be annotated and preset prompt words to generate annotation prompt words, and input the annotation prompt words into each of the large models in the initial mode pool to obtain multiple predicted annotations for the data to be annotated.

[0093] The voting output unit 303 is used to determine at least one output label from the plurality of prediction labels and output it to the real user based on a multi-model voting consensus decision-making mechanism.

[0094] The acquisition unit 304 is used to acquire user feedback from the real user regarding the output label; the user feedback includes an adoption status and a correction operation; the adoption status is used to indicate whether the output label has been adopted; the correction operation includes the adjustment content of the output label given by the real user, wherein the correction operation is empty when the adoption status indicates that the output label has been adopted;

[0095] The sample generation unit 305 is used to generate training samples by combining the user feedback and the data to be labeled.

[0096] The execution unit 306 is used to repeatedly execute the steps of responding to the data to be labeled, counting votes, and obtaining user feedback until a preset period is reached, and then stop, thus obtaining multiple training samples.

[0097] In one possible implementation, the sample generation unit 305 is specifically used for:

[0098] If the user feedback indicates that the output label is adopted, then the output label is applied to the data to be labeled to obtain the training sample;

[0099] If the user feedback indicates that the output label is not adopted, the output label is adjusted based on the correction operation to obtain an adjusted label, and the adjusted label is labeled on the data to be labeled to obtain the training sample.

[0100] In one possible implementation, the voting output unit 303 is specifically used for:

[0101] The large models in the initial model pool are obtained to vote on each prediction label based on the multi-model voting consensus decision mechanism;

[0102] Predicted labels with a number of votes greater than or equal to the consensus threshold are identified as output labels, and these output labels are output to the real users.

[0103] In one possible implementation, the device further includes:

[0104] A sample partitioning unit is used to divide the plurality of training samples into a test set and a training set;

[0105] The model optimization unit is used to determine the baseline model, and to perform supervised fine-tuning and reinforcement learning training on the baseline model using the training set to obtain the optimized model.

[0106] The model testing unit is used to test the optimized model using the test set to obtain the performance indicators of the optimized model.

[0107] The available model determination unit is used to determine the optimized model as an available model when the performance index of the optimized model meets the preset standard.

[0108] The deployment unit is used to replace the initial model pool with the available models and deploy them online to serve the real users.

[0109] In one possible implementation, the baseline model is selected from the initial model pool or external candidate models.

[0110] In addition, this application embodiment also provides a data annotation device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the data annotation method as described above.

[0111] In addition, this application embodiment also provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform the data annotation method described above.

[0112] This application's embodiments construct an initial model pool containing multiple large models from different sources and employ a multi-model voting consensus decision-making mechanism. This enables the rapid output of highly reliable annotation results without relying on manual pre-annotation of training data. This method not only significantly reduces reliance on manual annotation but also improves the accuracy of initial annotations through cross-validation between models. Furthermore, this application deeply couples user feedback with the data to be annotated, generating high-quality training samples. In this way, explicit corrections and implicit adoption behaviors in real-world usage scenarios can be transformed into structured labeled data. This not only synchronizes the annotation process with actual application scenarios, significantly improving data annotation efficiency, but also allows for continuous optimization of annotation quality based on accumulated real-world feedback.

[0113] The foregoing has provided a detailed description of a data annotation method, apparatus, device, and storage medium provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0114] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0115] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A data annotation method, characterized in that, The method includes: An initial model pool is constructed and deployed to serve real users; the initial model pool includes multiple large models from different sources. In response to the data to be labeled uploaded by the real user, the annotation prompt words are generated by combining the data to be labeled and the preset prompt words. The annotation prompt words are then input into each of the large models in the initial model pool to obtain multiple predicted annotations for the data to be labeled. Based on a multi-model voting consensus decision-making mechanism, at least one output label is determined from the multiple prediction labels and output to the real user. Obtain user feedback from real users regarding the output annotations, and generate training samples by combining the user feedback and the data to be annotated; the user feedback includes an adoption status and a correction operation; the adoption status is used to indicate whether the output annotation has been adopted; the correction operation includes the adjustment content of the output annotation given by the real users, wherein the correction operation is empty when the adoption status indicates that the output annotation has been adopted; Repeat the steps of responding to the labeled data, counting votes, and obtaining user feedback until the preset period is reached, then stop to obtain multiple training samples.

2. The method according to claim 1, characterized in that, The step of generating training samples by combining the user feedback and the data to be labeled includes: If the user feedback indicates that the output label is adopted, then the output label is applied to the data to be labeled to obtain the training sample; If the user feedback indicates that the output label is not adopted, the output label is adjusted based on the correction operation to obtain an adjusted label, and the adjusted label is labeled on the data to be labeled to obtain the training sample.

3. The method according to claim 1, characterized in that, The multi-model voting consensus decision-making mechanism determines at least one output label from the multiple prediction labels and outputs it to the real user, including: The large models in the initial model pool are obtained to vote on each prediction label based on the multi-model voting consensus decision mechanism; Predicted labels with a number of votes greater than or equal to the consensus threshold are identified as output labels, and these output labels are output to the real users.

4. The method according to claim 1, characterized in that, The method further includes: The multiple training samples are divided into a test set and a training set; A baseline model is determined, and the baseline model is then subjected to supervised fine-tuning and reinforcement learning training using the training set to obtain an optimized model. The optimized model is tested using the test set to obtain the performance indicators of the optimized model. When the performance indicators of the optimization model meet the preset standards, the optimization model is determined to be a usable model; The available models are used to replace the initial model pool and deployed online to serve the real users.

5. The method according to claim 4, characterized in that, The baseline model is selected from the initial model pool or external candidate models.

6. A data annotation device, characterized in that, The device includes: A deployment unit is built to construct an initial model pool and deploy the initial model pool to serve real users; the initial model pool includes multiple large models from different sources. The annotation unit is used to respond to the data to be annotated uploaded by the real user, combine the data to be annotated with preset prompt words to generate annotation prompt words, and input the annotation prompt words into each large model in the initial mode pool to obtain multiple predicted annotations for the data to be annotated. A voting output unit is used to determine at least one output label from the multiple prediction labels and output it to the real user based on a multi-model voting consensus decision-making mechanism. The acquisition unit is used to acquire user feedback from the real user regarding the output label; the user feedback includes an adoption status and a correction operation; the adoption status is used to indicate whether the output label has been adopted; the correction operation includes the adjustment content of the output label given by the real user, wherein the correction operation is empty when the adoption status indicates that the output label has been adopted; A sample generation unit is used to generate training samples by combining the user feedback and the data to be labeled. The execution unit is used to repeatedly execute the steps of responding to the data to be labeled, counting votes, and obtaining user feedback until a preset period is reached, at which point it stops and obtains multiple training samples.

7. The apparatus according to claim 6, characterized in that, The sample generation unit is specifically used for: If the user feedback indicates that the output label is adopted, then the output label is applied to the data to be labeled to obtain the training sample; If the user feedback indicates that the output label is not adopted, the output label is adjusted based on the correction operation to obtain an adjusted label, and the adjusted label is labeled on the data to be labeled to obtain the training sample.

8. The apparatus according to claim 6, characterized in that, The voting output unit is specifically used for: The large models in the initial model pool are obtained to vote on each prediction label based on the multi-model voting consensus decision mechanism; Predicted labels with a number of votes greater than or equal to the consensus threshold are identified as output labels, and these output labels are output to the real users.

9. A data annotation device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the data annotation method as described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the data annotation method as described in any one of claims 1-5.

Images