Man-machine collaborative data annotation method based on large model

Abstract

The invention discloses a man-machine collaborative data labeling method based on a large model. The method comprises the following steps: selecting representative and diversified samples by utilizing an active learning strategy; then, zero-sample or few-sample labeling is carried out on the selected samples in combination with a large language model, so that an initial labeling database is generated; and through a multi-round iteration mode combining automatic labeling and manual revision of the large language model, a labeling result is continuously optimized. According to the method provided by the invention, efficient starting can be realized under the condition that only a small amount of initial data is needed, the workload of manual labeling is remarkably reduced, and the accuracy and consistency of labeling results are improved.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and data annotation technology, and in particular relates to a human-computer collaborative data annotation method based on a large model. Background Technology

[0002] In recent years, artificial intelligence (AI) technology has developed rapidly, and various intelligent algorithms have been widely applied in multiple fields. Among them, Large Language Models (LLMs), with their massive parameter scale and powerful natural language understanding and generation capabilities, have become a prominent representative of AI research and applications. These models can play a role in various downstream tasks such as information extraction, text classification, question answering systems, dialogue generation, and data annotation, demonstrating superior performance in complex natural language processing.

[0003] In the field of data annotation, large models are increasingly being used to assist or replace manual annotation, thereby significantly reducing labor costs and accelerating the application of data in downstream tasks. However, relying solely on large language models for annotation still has certain limitations. On the one hand, models are prone to errors when processing domain-specific terminology, failing to correctly understand the meaning of technical terms and requiring explanation; on the other hand, large models have limitations in their ability to follow complex annotation rules, potentially resulting in lower annotation accuracy.

[0004] Patent document CN120319413A discloses an AI-based collaborative data annotation system, including a data preprocessing module, a multi-model collaborative annotation module, an AI server, a human-machine collaboration module, a quality assessment feedback module, and a scheduling module. The data preprocessing module includes acquiring basic sample data, processing the sample data, and outputting it to the multi-model collaborative annotation module to generate pre-annotation results. The scheduling module schedules the AI ​​server to use the human-machine collaboration module to perform precise standardization on the pre-annotation results based on the amount of sample data. The quality assessment feedback module acquires the precise annotation results, performs manual sampling inspection, and outputs the annotation results.

[0005] Patent document CN120086754A discloses a human-computer collaborative data annotation and cleaning method based on a multimodal large language model, comprising: obtaining text prompts for data annotation instructions and extracting text features of the text prompts using a text encoder; extracting data features in the data to be annotated that are associated with the annotated objects in the data annotation instructions using a feature extraction module; mapping the data features to a text space to obtain data feature text; inputting the text features and data feature text into a large language model to understand the data to be annotated and generating data annotation information corresponding to the text prompts for the data to be annotated; using a data annotation module to complete the data annotation of the data to be annotated and obtaining the algorithm annotation result; showing the algorithm annotation result to the staff and obtaining the staff's judgment on the accuracy of the algorithm annotation result; and cleaning the algorithm annotation result based on the judgment on the accuracy of the algorithm annotation result to obtain the data annotation result. Summary of the Invention

[0006] The purpose of this invention is to provide a human-computer collaborative data annotation method based on a large model. This method can achieve efficient startup with only a small amount of initial data, significantly reduce the workload of manual annotation, and improve the accuracy and consistency of annotation results.

[0007] To achieve the objectives of this invention, the following solution is provided: a human-computer collaborative data annotation method based on a large model, comprising the following steps: Step 1: Obtain the target dataset and use an active learning strategy to select a portion of the data from the target dataset to form an initial sample set; A large language model is used to perform zero-sample initial annotation on the initial sample set to obtain the initial annotation results corresponding to the initial samples. Subsequently, based on the RAG retrieval enhancement algorithm, the initial annotation results were iteratively optimized by utilizing the fact that the samples in the initial sample set are each other's few sample instances, in order to improve the annotation consistency of the initial sample set. Step 2: Using the current initial sample set as the retrieval source, for each piece of data to be processed, sort the remaining data in the initial sample set by similarity based on the RAG retrieval enhancement algorithm, and construct a small sample example set using the K samples with the highest similarity scores among the remaining data. The set of few sample examples is used as prompt words and input into the large language model for context learning. Based on the large language model after context learning, the data to be processed is re-annotated. The re-annotation results are revised manually, and the revised data is updated into the initial sample set, thus completing one round of human-machine joint annotation. Step 3: Repeat step 2 until the iteration termination condition is met to obtain the final labeled database.

[0008] This invention achieves dynamic expansion and quality optimization of the annotation database through a small-batch, multi-round collaborative iteration mechanism, and can flexibly combine active learning and retrieval-Augmented Generation (RAG) algorithms to significantly reduce the annotation startup cost.

[0009] Specifically, the active learning strategy refers to prioritizing samples with high information content and significant differences from the selected samples by calculating the representativeness and coverage of each sample.

[0010] Specifically, before constructing the initial sample set, the target dataset needs to be divided into batches and prioritized. The process is as follows: An active learning algorithm is used to calculate the representativeness and diversity indices of each sample, and a priority order is generated accordingly. Based on priority order, samples with adjacent order are divided into the same batch, forming a batch sequence; After batch division is completed, batches with higher priority are processed iteratively first.

[0011] Specifically, the implementation process of the active learning algorithm is as follows: Set the complete dataset as The subset has been labeled as Unlabeled subsets ; Active learning algorithms will work on unlabeled subsets Each candidate sample Conduct a labeling value assessment to determine its contribution to the expansion of the labeling database; Active learning algorithms select the samples with the highest labeled value and discard the unlabeled subset. Transfer to labeled subset Repeat this process until the size of the labeled subset reaches the expected size.

[0012] Specifically, the implementation process of the RAG retrieval enhancement algorithm is as follows: Let the initial set of labeled samples be... The input data is ; The retrieval process uses a similarity function to analyze the input data. With each initial sample Scoring is performed, and the results are sorted from highest to lowest score; Select the top results from the sorted list. These samples constitute a small sample instance set.

[0013] Specifically, the similarity function includes sparse vector representation and semantically dense vector representation.

[0014] Specifically, the termination conditions include: a. The number of manual revisions in the current round is insufficient to meet the requirements of a new round of effective updates, i.e., it is below the set minimum revision threshold; b. When the number of iterations in the current round reaches the upper limit defined by the exponential decay function, the system will forcibly terminate the iteration.

[0015] Specifically, the expression for the termination condition is as follows:

[0016] in, Indicates the round count within the current batch. ; Indicates the first The number of manual revisions made during the round; The minimum revision round threshold; This represents the difference between the initial number of rounds and the minimum number of rounds. It is the exponential decay constant; Indicates batch index; This indicates the minimum number of rounds.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Reduce initial annotation cost: Existing methods usually require at least hundreds of high-quality labeled samples as initial training data, while the present invention adopts a small batch and multiple rounds of iteration framework, with the smallest batch size as low as 5 samples, and only one batch is needed to start, thereby greatly reducing the initial manual input.

[0018] (2) Improve annotation accuracy: This invention reduces human annotation errors caused by fatigue through a human-machine collaborative approach using a large model, while simultaneously compensating for the lack of professional knowledge in the large model through human revision and correcting its misunderstandings. By combining the consistency of artificial intelligence with the professional knowledge of humans, the overall accuracy of the annotation database is effectively improved. Attached Figure Description

[0019] Figure 1 This is a flowchart of the human-computer collaborative data annotation method based on a large model provided in this embodiment; Figure 2 This is a visualization of the impact of different annotation methods provided in this embodiment on the knowledge extraction performance of large models. Figure 3 This is a visualization of the impact of different annotation batch sizes and annotation data volumes on the final knowledge extraction effect of the large model, as provided in this embodiment. Figure 4 This is a visualization of the manual revision costs during annotation provided in this embodiment; Figure 5This is a visualization of the manual inspection cost during annotation provided in this embodiment. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0021] This embodiment provides a human-computer collaborative data annotation method based on a large model, including the following steps: (1) Cold start annotation. First, a subset of samples is selected from the complete dataset to construct an initial annotation database. A large language model is used to perform initial annotation on these samples, generating initial annotation results. Based on this, the large language model is used again for multiple rounds of iterative annotation to optimize the initial annotation results; (2) Batch partitioning. After the initial labeled database is generated in step (1), the labeled samples are evaluated and sorted according to their labeling priority. Then, the entire database is divided into several small batches according to the order of sample priority; (3) Multiple iterations. Within each batch, the annotation results are first updated using a large language model, and then human annotators revise the updated results, thus forming a complete human-machine joint annotation round. Each batch can undergo multiple iterations. After each batch iteration is completed, the final results are overwritten into the annotation database.

[0022] The cold start in step (1) is implemented as follows: a. In the initial stage of the method, samples that balance representativeness and diversity are first selected from the complete dataset to construct an initial labeled database. The selection of samples adopts an active learning strategy, which calculates the representativeness and coverage of each sample, and prioritizes samples with high information content and large differences from existing samples; b. Subsequently, combining the prompt word engineering method, the large language model is used to perform initial labeling on the samples selected by the active learning strategy in step a under the condition of no example input; c. After the initial annotation is completed, the initial annotation database generated in step b is iteratively updated multiple times using the RAG retrieval enhancement algorithm and a large language model. In each iteration, based on the RAG retrieval enhancement algorithm, the annotation most similar to the sample to be updated is selected from the existing annotation database as an example. The retrieved examples, combined with the prompt word input, utilize the context learning capability of the large language model to re-annotate the sample to be updated. After each iteration, the updated annotation results are written to the annotation database.

[0023] The specific implementation method for batch division and priority sorting in step (2) is as follows: a. In the cold-start annotation database obtained in step (1), the annotation priority of the samples is re-evaluated and sorted. The evaluation adopts an active learning algorithm to calculate the representativeness index and diversity index of each sample, and generate a priority order accordingly; b. Based on the priority order obtained in step a, group adjacent samples into the same batch, forming a batch sequence. The batch size can be adjusted according to the optimal configuration of the RAG algorithm; c. After batch division is completed, high-priority batches are processed iteratively first to quickly improve the quality of the overall annotation database.

[0024] The specific implementation method for the batch-based multi-round iterative optimization in step (3) is as follows: a. An example selection method based on RAG retrieval enhancement selects examples from an existing labeled database, combines them with prompt word input, and uses a large model to re-label all samples in the batch; b. After the model annotation results are generated, they are revised by human annotators. Human revisions can be a full review or only on parts of the model with high uncertainty or rule conflicts. The revised results, together with the model annotation results, constitute one round of human-machine joint annotation for this batch, and the updated annotation results are written into the annotation database. c. Within the same batch, steps a and b are repeated to form a multi-round human-machine joint iterative process. Each iteration starts based on the update result of the previous round, until the termination condition is met; d. When the termination condition is triggered, the human-machine joint iteration process for this batch ends, and the final revision result is updated and written to the annotation database, completing the update for this batch.

[0025] The termination condition for human-machine joint annotation in step (3) is implemented as follows: The number of manual revisions in the current round is insufficient to meet the requirements of a new round of effective updates, i.e., it is below the set minimum revision threshold; When the number of iterations in the current round reaches the upper limit defined by the exponential decay function, the system will forcibly terminate the iteration. This upper limit gradually decreases with the batch index, thereby avoiding over-iteration.

[0026] The termination condition can be formally expressed as:

[0027] in, Indicates the round count within the current batch. ; Indicates the first The number of manual revisions made during the process; Minimum revision threshold (default value is 0); This represents the difference between the initial number of rounds and the minimum number of rounds (default value is 5). This is the exponential decay constant (default value is 0.1). Indicates batch index; This indicates the minimum number of rounds (default value is 1).

[0028] Termination conditions are not limited to the above methods and can be set according to actual task requirements, such as based on annotation consistency or performance indicators.

[0029] The active learning in steps (1) and (2) is specifically implemented as follows: Let the complete dataset be The subset has been labeled as Unlabeled subsets Active learning algorithms will work on unlabeled subsets. Each candidate sample Conduct a labeling value assessment to determine its contribution to the expansion of the labeling database.

[0030] The labeled value of each candidate sample is calculated using the following formula:

[0031] in, This is a similarity measurement function between samples. Indicates the size of the unlabeled sample set. This is the balancing parameter (default value is 0.5).

[0032] The first part of the above formula measures the sample. The first part ensures the representativeness of the unlabeled set, guaranteeing that it reflects the overall characteristics of the remaining data; the second part penalizes candidates that are highly similar to the labeled samples, ensuring sample diversity.

[0033] Active learning algorithms select the samples with the highest labeled value and discard the unlabeled subset. Transfer to labeled subset Repeat this process until the size of the labeled subset reaches the expected size, and then proceed to the subsequent labeling process.

[0034] The RAG retrieval enhancement algorithm in steps (1) and (3) is specifically implemented as follows: Let the annotation database be The input data is When performing automatic annotation, the RAG algorithm retrieves the most relevant data from the database. Here is an example as a reference for generating annotations using a large language model. The size can be flexibly set according to the database size and task complexity.

[0035] The retrieval process uses a similarity function For input data With database samples Scoring is performed, and the results are sorted from highest to lowest score.

[0036] Then, select the top from the sorted results. These samples constitute a small sample set:

[0037] This set, as contextual examples, is fed into the large language model along with the input paragraph to generate new annotation results.

[0038] The similarity function can be implemented in two ways, including but not limited to: one based on sparse vector representation (such as TF-IDF, BM25); and the other based on semantically dense vector representation (such as BERT, SBERT). Different similarity calculation methods can be selected according to the database size and task characteristics.

[0039] More specifically, such as Figure 1 The diagram illustrates the application of this embodiment in the synthesis path extraction task of MOFs (Metal-Organic Frameworks).

[0040] In this embodiment, the MOF literature data comes from the Cambridge Structural Database (CSD), which covers 80,000 MOF data points. 440 articles were randomly selected from this database, their synthetic paragraphs were annotated, and a BERT model was trained to identify these synthetic paragraphs. Subsequently, a portion of articles were randomly selected from the entire dataset, the PDF files were converted to TXT files, and the corresponding synthetic paragraphs were identified from these TXT files to form a synthetic paragraph database.

[0041] Meanwhile, by utilizing the prompt word engineering, we combine role settings, task descriptions, background knowledge, annotation rules, and output formats to generate prompt words for synthetic condition extraction.

[0042] After obtaining the required data and prompt words, the first step is cold start annotation. From the complete synthetic paragraph database, 123 representative and diverse synthetic paragraphs are selected using an active learning strategy. In the initial annotation, the large language model performs zero-sample sampling on these paragraphs to generate synthetic condition labels including metal precursors, organic ligands, solvents, regulators, reaction temperatures, and reaction times. Subsequently, a few-sample learning strategy is used for multiple iterations of optimization. The RAG retrieval enhancement algorithm is applied to select four similar paragraphs and their corresponding initial annotations for each synthetic paragraph. Combined with prompt words, the large model is guided to further optimize the annotation through contextual learning. Furthermore, through multiple iterations, the accuracy and consistency of cold start annotation are continuously improved.

[0043] The RAG algorithm is implemented using the BM25 sparse vector representation method. BM25 is a probabilistic information retrieval model that ranks documents based on the frequency of query terms within the document. It balances term frequency (the number of times a word appears in a document) with inverse document frequency (the rarity of a word in the entire document set), thus giving higher weight to more meaningful words.

[0044] BM25 for a containing A paragraph of words and a size of A document in a collection of documents The scoring function between them is defined as follows:

[0045]

[0046] in Indicator In the document word frequency in Document Length, This represents the average length of all documents in the document collection. Indicator Inverse document frequency. and These are the hyperparameters of the model; the default settings for BM25 were used in the experiment. , .

[0047] After generating the initial labeled database during cold start, step (2) batch partitioning is performed. The existing 123 synthetic paragraphs are re-prioritized and divided into several smaller batches. Based on the optimal results of the RAG retrieval enhancement algorithm on CSD (the best performance is achieved when retrieving 4 examples for context learning in the case of BM25 similarity), each batch can contain 5 samples. This strategy can reduce the burden of manual annotation and enable the subsequent model to continuously learn and optimize.

[0048] After batch division, each batch is iterated in multiple rounds according to priority from high to low. In each small batch, the RAG retrieval enhancement algorithm is first used to select similar examples from the cold start database based on BM25 similarity, and the large model is guided to annotate the text of this batch by combining prompt words; then, human annotators revise the model output item by item to correct the model's errors in chemical entity classification, unit recognition, etc. For example, the model may misidentify "heat" and "pure" as chemical substance names, and human revision can eliminate such errors. The revised results and the model output together form the first round of human-machine collaborative annotation results for this batch. This process can be iterated multiple times until the number of human revisions is lower than the threshold or the preset number of rounds is reached. Finally, the annotation results of this batch are written into the database to update the subsequent model input.

[0049] To verify the improvement in labeled data quality achieved by the method of this invention, this embodiment uses different labeling methods for data labeling and uses them as a sample library for RAG retrieval enhancement to test the accuracy of extracting 123 synthetic paragraphs. The following three methods are used as test baselines: 1. No labeling is used, only synthetic paragraphs are given; 2. Data is labeled only by the large model; 3. Data is labeled only by humans.

[0050] like Figure 2 The image shown is a visualization of the impact of different annotation methods on the knowledge extraction performance of large models. Figure 2 It can be seen that the solution provided in this embodiment significantly improves the RAG extraction performance of large models in terms of F1 and ACC metrics compared to the three baseline methods mentioned above.

[0051] This embodiment also tested the impact of batch size on the conditional extraction task of MOF synthesis for large models. The changes in synthesis conditional extraction performance with increasing labeled data volume were tested for batch sizes of 5, 15, and 40. Figure 3 It can be seen that, with different batch size settings, the extraction performance of large models can be significantly improved when labeling 0% to 30% of the data. At the same time, when the batch size is set to 5, this method can achieve near-optimal human labeling performance (about 99%) after labeling only about 44.7% of the samples.

[0052] To verify the effectiveness of the method of the present invention in reducing the cost of labeled data, this embodiment statistically analyzes and compares the revision and inspection costs required for human-computer collaborative labeling and other labeling methods on 123 data points.

[0053] Revise the cost verification results as follows Figure 4 As shown, revision cost is defined as the number of synthetic conditions that require manual revision of the previous round of labeled data. Compared to purely manual annotation, the human-machine collaborative data annotation method based on large models reduces the number of revisions by 82.4%–90.8%. Compared to the improved baseline that uses zero-sample large model pre-annotation and performs one revision, this method can further reduce revision costs by 25.3%–60.9%. In particular, when using the smallest batch size, the amount of manual revision is minimized, and the annotation accuracy improves more rapidly.

[0054] Check the cost verification results as follows Figure 5 As shown. The inspection cost is defined as the amount of work required to manually check the correctness of the previous round of annotations before attempting a revision. Validation results show that the minimum batch setting also results in the lowest scanning cost. Further evidence shows that under this setting, the number of iterations required per batch decreases more rapidly, further reducing the cost of annotated data.

[0055] In summary, this embodiment verifies that the method of the present invention can obtain near-optimal annotation quality with relatively low manual cost, which meets the design goals of high efficiency and low cost.

[0056] Furthermore, the terms "upper," "lower," "inner," "outer," "front," and "rear" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0057] Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of the present invention. All equivalent changes or modifications made to the structure, features and principles described in the claims of the present invention should be included in the scope of the claims of the present invention.

[0058] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A human-computer collaborative data annotation method based on a large model, characterized in that, Includes the following steps: Step 1: Obtain the target dataset and use an active learning strategy to select a portion of the data from the target dataset to form an initial sample set; A large language model is used to perform zero-sample initial annotation on the initial sample set to obtain the initial annotation results corresponding to the initial samples. Subsequently, based on the RAG retrieval enhancement algorithm, the initial annotation results were iteratively optimized by utilizing the fact that the samples in the initial sample set are each other's few sample instances; Step 2: Using the current initial sample set as the retrieval source, for each piece of data to be processed, sort the remaining data in the initial sample set by similarity based on the RAG retrieval enhancement algorithm, and construct a small sample example set using the K samples with the highest similarity scores among the remaining data. The set of few sample examples is used as prompt words and input into the large language model for context learning. Based on the large language model after context learning, the data to be processed is re-annotated. The re-annotation results are revised manually, and the revised data is updated into the initial sample set, thus completing one round of human-machine joint annotation. Step 3: Repeat step 2 until the iteration termination condition is met to obtain the final labeled database.

2. The human-computer collaborative data annotation method based on a large model according to claim 1, characterized in that, The active learning strategy refers to prioritizing samples with high information content and significant differences from the selected samples by calculating the representativeness and coverage of each sample.

3. The human-computer collaborative data annotation method based on a large model according to claim 1, characterized in that, Before constructing the initial sample set, the target dataset needs to be batch-split and prioritized. The process is as follows: An active learning algorithm is used to calculate the representativeness and diversity indices of each sample, and a priority order is generated accordingly. Based on priority order, samples with adjacent order are divided into the same batch, forming a batch sequence; After batch division is completed, batches with higher priority are processed iteratively first.

4. The human-computer collaborative data annotation method based on a large model according to claim 3, characterized in that, The implementation process of the active learning algorithm is as follows: Set the complete dataset as The subset has been labeled as Unlabeled subsets are ; Active learning algorithms will work on unlabeled subsets Each candidate sample Conduct a labeling value assessment to determine its contribution to the expansion of the labeling database; Active learning algorithms select the samples with the highest labeled value and discard the unlabeled subset. Transfer to labeled subset Repeat this process until the size of the labeled subset reaches the expected size.

5. The human-computer collaborative data annotation method based on a large model according to claim 1, characterized in that, The implementation process of the RAG retrieval enhancement algorithm is as follows: Let the initial set of labeled samples be... The input data is ; The retrieval process uses a similarity function to analyze the input data. With each initial sample Scoring is performed, and the results are sorted from highest to lowest score; Select the top results from the sorted list. These samples constitute a small sample instance set.

6. The human-computer collaborative data annotation method based on a large model according to claim 5, characterized in that, The similarity function includes sparse vector representation and semantically dense vector representation.

7. The human-computer collaborative data annotation method based on a large model according to claim 1, characterized in that, The termination conditions include: a. The number of manual revisions in the current round is insufficient to meet the requirements of a new round of effective updates, i.e., it is below the set minimum revision threshold; b. When the number of iterations in the current round reaches the upper limit defined by the exponential decay function, the system will forcibly terminate the iteration.

8. The human-computer collaborative data annotation method based on a large model according to claim 1 or 7, characterized in that, The expression for the termination condition is as follows: ;in, Indicates the round count within the current batch. ; Indicates the first The number of manual revisions made during the round; The minimum revision round threshold; This represents the difference between the initial number of rounds and the minimum number of rounds. It is the exponential decay constant; Indicates batch index; This indicates the minimum number of rounds.

Images