Large language model, data programming and active learning driven text labeling method and system
By combining data programming with active learning and utilizing large language model analysis, the problem of balancing efficiency and cost in text annotation was solved, realizing an efficient and iterative text data annotation process that improved annotation accuracy and coverage.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-04
- Publication Date
- 2026-03-13
AI Technical Summary
Existing technologies struggle to balance efficiency, accuracy, and cost in the field of text annotation. Data programming is fast but limited in accuracy, active learning suffers from the cold start problem, and large language models are computationally expensive and restrict large-scale applications.
By combining data programming and active learning, efficient text annotation is achieved through attribute configuration, labeling functions, active learning algorithms, and large language model analysis.
It improves annotation efficiency and accuracy, reduces user workload, supports iterative construction and reuse of annotation functions, adapts to different scenario needs, and has flexibility and practicality.
Smart Images
Figure CN121659909A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of human-computer interaction and natural language processing technology, and in particular to a text annotation method and system driven by large language models, data programming, and active learning. Background Technology
[0002] Deep learning models can achieve excellent results in natural language processing tasks through supervised training on large-scale text annotation datasets. However, constructing large-scale, high-quality text annotation datasets often requires significant manpower and funding, making large-scale, efficient, and accurate annotation a key element in the development of deep learning. Among existing data annotation methods, data programming allows users to write labeling functions to automatically annotate data. Data programming offers advantages such as speed and low cost, but in practical applications, it is often limited by the accuracy and coverage of annotations. Furthermore, designing efficient and reliable labeling functions is a challenging task for annotators who are not proficient in programming. Active learning methods select data with the highest uncertainty (i.e., the highest information gain) for annotators to label, aiming to improve model performance with minimal annotation costs. However, although active learning can improve annotation accuracy to some extent, its performance is susceptible to the cold-start problem, meaning that when initial labeled data is insufficient, sample selection may not be optimal, thus limiting the model's learning effect. In recent years, large language models have demonstrated the potential to achieve high-precision and flexible annotation with minimal human intervention, significantly reducing the workload of manual annotation. However, due to the high computational demands and corresponding economic costs of their training and inference processes, these methods still face significant limitations in large-scale deployment. Secondly, building labeled datasets typically involves an iterative process encompassing active learning and data programming, which still requires substantial user involvement, making it time-consuming and hindering overall efficiency. These factors make the construction of task-specific object sentiment analysis datasets time-consuming and labor-intensive.
[0003] Current annotation tools still struggle to balance efficiency, accuracy, and cost. Data programming is fast and inexpensive, but limited by accuracy and coverage, and non-programmers find it difficult to write effective annotation functions. Active learning can improve accuracy, but suffers from a cold start problem. Large language models provide accurate annotations, but their computational and financial costs are too high, limiting large-scale application. To address the difficulty of balancing efficiency and cost in text annotation, this invention proposes a text annotation method and system driven by large language models, data programming, and active learning. Summary of the Invention
[0004] The purpose of this invention is to provide a text annotation method and system driven by large language models, data programming, and active learning. This method combines data programming and active learning to achieve large-scale data annotation and improves annotation efficiency by leveraging large language models.
[0005] To achieve the above objectives, the present invention provides the following solution:
[0006] A text annotation method driven by large language models, data programming, and active learning includes:
[0007] Configure attributes for the current annotation task to obtain the definition of the current annotation task, wherein the attributes include task type, label target, and label category;
[0008] According to the definition of the current annotation task, a tagging function is configured. The tagging function tags keywords through a word set, maps the word set to candidate tag categories using heuristic rules, and aggregates the candidate tag categories to obtain the most suitable tag for the text using an aggregation method.
[0009] An active learning algorithm is used to select the sample with the highest uncertainty, wherein the uncertainty is obtained through model uncertainty sampling, label function uncertainty sampling, and unlabeled samples;
[0010] By analyzing the selected samples using a large language model, we can achieve word set expansion, tag function recommendation, and sample analysis.
[0011] Optionally, the task types include text classification tasks and target-specific annotation tasks;
[0012] The labeling target is used to specify the labeling target for the specific target labeling task;
[0013] The tag category is used to define the set of tags available for the task.
[0014] Optionally, the heuristic rules are used for direct mapping of a single word set to a label category, as well as sequential mapping of multiple word sets in combination.
[0015] Optionally, aggregating the candidate tag categories using an aggregation method includes:
[0016] The nearest neighbor method is used to determine the position of the target word in the text and search for the nearest tag category;
[0017] Alternatively, a window analysis method can be used to set a radius range centered on the target, and generate an overall label within a local window through majority voting.
[0018] Optionally, the model uncertainty sampling includes: using the data generated by the labeling function to train a pre-trained model to obtain the semantic relationship between different data objects, generating the prediction probability of each data, and selecting the sample with the lowest confidence based on the prediction probability;
[0019] The uncertainty sampling of the labeling function selects samples based on the divergence of multiple labeling functions through a committee query algorithm.
[0020] Optionally, the word set expansion includes: analyzing the selected samples through the large language model and automatically identifying words related to each word set to achieve word set expansion.
[0021] This invention also provides a text annotation system driven by large language models, data programming, and active learning, comprising:
[0022] The labeling task definition module is used to configure the attributes of the current labeling task and obtain the definition of the current labeling task. The attributes include task type, labeling target and label category.
[0023] The tagging function configuration module is used to configure the tagging function according to the definition of the current tagging task. The tagging function tags keywords through a word set, maps the word set to candidate tag categories using heuristic rules, and aggregates the candidate tag categories to obtain the most suitable tag for the text using an aggregation method.
[0024] An active learning sampling module is used to select the sample with the highest uncertainty using an active learning algorithm, wherein the uncertainty is obtained through model uncertainty sampling, label function uncertainty sampling, and unlabeled samples;
[0025] The large language model analysis module is used to analyze selected samples using a large language model, enabling word set expansion, tag function recommendation, and sample analysis.
[0026] The beneficial effects of this invention are as follows: This invention proposes a structured tagging function creation syntax for text data annotation, enabling users and large models to define tagging rules through configuration and automatically convert them into executable tagging functions by the system. This invention first rapidly generates a large number of text labels using tagging functions, and then uses an active learning strategy to select the tagging samples with the highest uncertainty (i.e., the highest information gain). Large language models automatically analyze these samples and generate tagging functions using the tagging function creation syntax defined in this invention. The constructed tagging functions support iterative construction and reuse in similar annotation tasks.
[0027] This invention's system achieves an efficient, iterative text data annotation workflow by combining data programming, active learning, and large language model assistance. Based on this workflow, an interactive system is implemented. Users can easily select samples in instance analysis and distribution views, extract heuristic rules, and quickly construct labeling functions. The system supports flexible combination and configuration of rules and achieves automatic annotation and continuous optimization of the dataset through fine-tuning aggregation. During this process, the active learning view guides users to focus on samples with higher information content through visualization, promoting the continuous improvement of labeling functions. The large language model assistance function effectively reduces the workload of users in word annotation, automatically expanding the word set and recommending labeling functions, not only improving coverage and effectiveness but also helping users make more accurate annotation decisions when dealing with complex samples such as irony and idioms. Overall, this invention's system significantly improves annotation efficiency and accuracy, combining flexibility and practicality, allowing users to quickly get started and use it in a short period of time. Furthermore, the various functional views of this invention are closely connected, forming a complete and coherent workflow, which effectively supports efficient and smooth data annotation tasks. This makes this invention not only applicable to general scenarios, but also able to flexibly meet the needs of individual developers to build datasets in specific fields, demonstrating strong practicality and adaptability. Attached Figure Description
[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0029] Figure 1 This is a flowchart illustrating a text annotation method driven by a large language model, data programming, and active learning, according to an embodiment of the present invention.
[0030] Figure 2 This is a schematic diagram of the tagging function template according to an embodiment of the present invention;
[0031] Figure 3 A flowchart illustrating the annotation process of the tagging function created for an embodiment of the present invention;
[0032] Figure 4 This is a flowchart illustrating how the large language model analysis module of this invention fully supports the annotation process by performing the following tasks;
[0033] Figure 5 This is a schematic diagram illustrating how the large language model generates prompts for three tasks according to an embodiment of the present invention;
[0034] Figure 6A schematic diagram illustrating the creation of a page for a marking function in the system interactive interface of this invention embodiment;
[0035] Figure 7 This is a schematic diagram of the active learning distribution page in the system interaction interface of this invention embodiment;
[0036] Figure 8 This is a schematic diagram of the large model auxiliary page in the system interaction interface of this invention embodiment;
[0037] Figure 9 Create a view for the tagging function in the embodiments of the present invention;
[0038] Figure 10 This is a word set selection view according to an embodiment of the present invention. Detailed Implementation
[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0040] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0041] This embodiment proposes a text annotation method driven by a large language model, data programming, and active learning, including:
[0042] Configure the attributes of the current annotation task and obtain the definition of the current annotation task. The attributes include task type, label target and label category.
[0043] Configure the tagging function according to the definition of the current annotation task. The tagging function tags keywords through a word set, maps the word set to candidate label categories using heuristic rules, and aggregates the candidate label categories to obtain the most suitable label for the text using an aggregation method.
[0044] An active learning algorithm is used to select the sample with the highest uncertainty. The uncertainty is obtained through model uncertainty sampling, label function uncertainty sampling, and unlabeled samples.
[0045] By analyzing the selected samples using a large language model, we can achieve word set expansion, tag function recommendation, and sample analysis.
[0046] Furthermore, the task types include text classification tasks and target-specific annotation tasks;
[0047] Label targets are used to specify labeling targets for a specific target labeling task;
[0048] Tag categories are used to define the set of tags available for a task.
[0049] Specifically, (1) the labeling task definition includes:
[0050] Users need to define the current annotation task by configuring three core attributes: task type, annotation target, and label category. These three core attributes collectively determine the target of the annotation function and clarify the nature of the generated dataset. These attributes define the granularity of the dataset, specify the set of labels included, and the semantic meaning of each label in the data, thereby ensuring that all annotation functions generate consistent and interpretable annotation results.
[0051] (a) Task Type This attribute clarifies the granularity of the annotation task. In this embodiment, two task types with different granularities are provided to the user: text classification tasks and target-specific annotation tasks. The goal of text classification tasks is to assign a single label to the entire text, while target-specific annotation tasks focus on annotating specific objects in the text, thereby achieving more refined analysis.
[0052] (b) Target. For specific target annotation tasks, users need to specify the annotation target and can set target aliases to improve the recognition ability of the labeling function, while text classification tasks do not require this attribute.
[0053] (c) Tag Categories. Tag categories define the set of tags available for a task. This invention supports flexible customization, allowing users to select the number and type of tags according to specific task requirements, thereby adapting to the annotation requirements of different scenarios.
[0054] Furthermore, heuristic rules are used for direct mapping of a single word set to a label category, as well as sequential mapping of multiple word sets in combination.
[0055] Furthermore, the aggregation of candidate tag categories using aggregation methods includes:
[0056] The nearest neighbor method is used to determine the position of the target word in the text and search for the nearest tag category;
[0057] Alternatively, a window analysis method can be used to set a radius range centered on the target, and generate an overall label within a local window through majority voting.
[0058] Specifically, the tag function configuration includes:
[0059] After completing the task definition, this embodiment provides users with a standardized syntax for configuring tagging functions, enabling them to quickly build runnable tagging functions based on their own heuristic rules. This syntax first inherits the three attributes of task type, target, and tag category from the tagging task definition above, allowing for a unified task definition for different created tagging functions. Secondly, users can customize different tagging rules in each tagging function by combining input word sets and selecting corresponding tag categories. Each tagging rule is defined in the form of [tag category: word set combination]. The word set involved in the rule is automatically read by the system and passed to the tagging function to assist in tagging specific words in the text. Finally, this syntax allows users to select different aggregation methods based on task characteristics to aggregate multiple tag categories transformed by the rules and return a unique tag category as the label for the sample being annotated. Functions created based on this template first extract keywords and annotate words in the text that exist in the corresponding word sets; then, they map these word sets to candidate tag categories using heuristic rules; finally, they aggregate the candidate tags using aggregation methods to determine the most suitable tag for the text.
[0060] (a) Lexicon. Text can be categorized into a specific label because certain parts of it exhibit characteristics of that label. This embodiment uses a predefined lexicon to capture these key features. In text processing, a lexicon represents a set of semantically related words; for example, a lexicon for "support" might include words like "trust," "believe," "agree," and "support." The tagging function tags keywords in the text using the lexicon, thereby extracting key features.
[0061] (b) Heuristic Rules. After capturing key features using word sets, this embodiment maps them to specific label categories using heuristic rules. Heuristic rules are used to map one or more word sets to specific label categories. Users can flexibly define the mapping relationship between word sets and label categories according to the classification task. For example, in a stance detection task, if the label categories are "support" and "oppose," the user can create a word set representing the semantics of support, which may contain words such as "agree" and "promise," and a word set representing the semantics of opposition, which may contain words such as "criticize" and "reject." Then, rules are defined to map the word set representing the semantics of support to the "support" label, and the word set representing the semantics of opposition to the "oppose" category. In this way, the labeling function can identify words in the text that carry stance, thereby improving the overall classification accuracy.
[0062] In more complex contexts, simple mappings between single word sets and labels may be insufficient to capture semantics. For example, when there are logical relationships or linguistic dependencies between multiple keywords, rules need to support sequential mappings, that is, considering multiple word sets together to determine the correct label category. For instance, the sentence "I disagree with Smith being president" clearly expresses opposition. We can label "no" as a negative word and "agree" as a supporting word, and then define a rule: when a negative word appears before a supporting word, the combination expresses an "opposition" stance. This demonstrates the polarity reversal effect of negative words when sequentially connected with supporting cues, allowing the text to be correctly classified as "opposition." More broadly, this example shows how context-sensitive interactions between linguistic features can trigger semantic changes that affect accurate classification.
[0063] (c) Aggregation Method. As mentioned earlier, after text is abstracted into a series of label categories through rule-based mapping, each label category represents the classification result of a local text. Next, an aggregation method is used to merge these local label categories into a single overall label as the final text annotation. This specification provides a text analysis-based aggregation method, which generates an overall label by statistically analyzing the identified label categories and using a majority voting principle. This approach can mitigate the influence of locally biased labels and improve generalization ability across different text classification tasks. However, in tasks requiring fine-grained attention to specific targets, unlike text classification tasks, the global context may introduce irrelevant information. To address this issue, this specification proposes two annotation methods that prioritize the local context of the target object. One is the nearest neighbor method, which first determines the position of the target word in the text, then searches for the nearest label category, and can adapt to different language structures through search direction parameters. The other is the window analysis method, which sets a radius around the target and generates an overall label within this local window using a majority voting principle. Both methods focus on words near the target, capturing relevant relationships as much as possible and reducing noise from irrelevant text.
[0064] After users create multiple labeling functions, a generative model is used to integrate the outputs of these functions and infer a unique label for each sample. The word set and labeling functions generated in this task can be seamlessly transferred to similar tasks, improving reusability and reducing repetitive work.
[0065] Furthermore, model uncertainty sampling includes: using the data generated by the labeling function to train a pre-trained model to obtain semantic relationships between different data objects, generating the prediction probability of each data, and selecting the sample with the lowest confidence based on the prediction probability;
[0066] Uncertainty sampling of labeling functions selects samples based on the divergence of multiple labeling functions through a committee query algorithm.
[0067] Specifically, active learning sampling includes:
[0068] The labels selected for each text by the labeling function often contain noise and require further optimization. To address this issue, this embodiment employs an active learning algorithm to identify samples with the highest uncertainty (i.e., the most informative value) for further labeling. By focusing attention on key samples, the system can correct erroneous labels during the iteration process, reduce redundant work caused by creating overlapping labeling functions, and ultimately improve the accuracy of the overall labeling process. The active learning algorithm can generate diverse sampling results using different sources of uncertainty, thereby providing users with multi-faceted sample exploration and analysis. This embodiment considers three types of uncertainty sources: model uncertainty sampling, labeling function uncertainty sampling, and unlabeled sampling.
[0069] (a) Model Uncertainty Sampling. This invention utilizes data generated by a labeling function to train a RoBERTa pre-trained model to capture semantic relationships between different data objects, thereby generating a prediction probability for each data point. Based on this, a "lowest confidence sampling" method is employed to select a batch of samples with the lowest confidence in the model's predictions. These samples are often the most difficult for the model to distinguish, thus providing the greatest information gain for further annotation.
[0070] (b) Uncertainty Sampling of Labeling Functions. Since different labeling functions are created based on different rules, they may produce different results for the same data, and their discrepancies can also be a significant source of uncertainty. Therefore, this embodiment draws on the idea of a "committee query" algorithm: by maintaining a set of differentiated models (in this embodiment, multiple labeling functions created by the user), candidate samples are voted on, and the samples with the greatest discrepancy and the most controversy are ultimately selected for the next round of labeling. These samples are considered to contain the most valuable information.
[0071] (c) Unlabeled Sampling. Unlabeled samples refer to data that has not been classified into existing categories by any labeling function. These samples typically contain features not yet covered by current labeling rules. By analyzing and utilizing these samples, users can effectively expand labeling rules, thereby improving overall coverage. Therefore, this invention incorporates the querying of fuzzy samples into an active learning strategy to continuously improve labeling quality.
[0072] Furthermore, expanding the vocabulary set involves analyzing the selected samples using a large language model and automatically identifying words related to each vocabulary set.
[0073] Specifically, large language model-assisted analysis includes:
[0074] This embodiment utilizes a large language model to provide multifaceted support for the annotation process. Specifically, the invention automatically analyzes the data samples selected by the active learning module, extracts appropriate words for each word set, and generates corresponding annotation functions based on the grammar configuration proposed in this invention. Furthermore, for some complex or difficult-to-process samples, this module can provide analytical insights to help users correct incorrect labels and inspire users to generate new heuristic rules to improve the annotation functions.
[0075] (a) Lexicon Expansion. As mentioned earlier, the effectiveness of the tagging function in this embodiment highly depends on word tagging, but manually tagging all keywords would place a significant burden on users. To address this issue, this embodiment uses a large language model to analyze the sampled data and automatically identify words associated with each lexicon. Users only need to tag a few examples for the large language model to infer the semantic meaning of each lexicon. Once the lexicon is defined, the large language model can automatically detect and extract the corresponding words from the samples.
[0076] (b) Tagging Function Recommendation. To enhance the effectiveness of tagging functions, this embodiment analyzes sampled data, attempts to extract universally applicable heuristic rules, and selects a suitable aggregation method for the current task, thereby recommending tagging functions based on the tagging function configuration syntax. The returned tagging functions are provided in JSON format and can be directly converted into executable code. By automating the discovery and formulation of heuristic rules, this method not only reduces the user's manual workload but also improves the generalization ability of tagging functions across different tasks.
[0077] (c) Sample Analysis. This embodiment automatically analyzes samples selected through an active learning strategy and uses a large language model to generate recommended tags and their detailed explanations. This process fully leverages the contextual reasoning capabilities of the large language model, enabling it to handle complex situations that word-based tagging functions struggle with, such as ironic or idiom expressions, thereby improving tagging accuracy. Furthermore, by providing explicit explanations, this module can inspire users to design new heuristic rules, which can then be used to optimize and refine existing tagging functions. The data sampled through active learning is also analyzed by the large language model analysis module. The analysis results are displayed in the large language model analysis view. This module automatically extracts words suitable for the current word tag from the samples and allows users to add them to the word set as needed. Additionally, the large language model learns heuristics from these samples, which users can refer to or directly adopt from the tagging function templates it provides.
[0078] like Figure 1As shown, given an unlabeled dataset, users can create a labeling function to automatically annotate it. This embodiment uses an active learning algorithm to select samples and provides a large language model for auxiliary analysis, helping users extract heuristics and refine the labeling function. The goal of this embodiment is to improve annotation quality through iteration and interaction. Finally, the labeled dataset is output. The flowchart includes the dataset, labeling function configuration, active learning adoption, and large language model analysis during the processing.
[0079] like Figure 2 The tagging function defined according to the tagging function template consists of two parts: task definition and function configuration. The task definition clarifies the task type and the set of tag categories used throughout the tagging process; the function configuration assigns specific logic to each function to guide the tag generation process. This specification uses a progressive abstraction approach to determine tags for text data. Functions created according to this specification first extract keywords and tag them using word sets, then use heuristic rules to map these word sets to candidate tag categories. Finally, an aggregation method is used to aggregate the candidate tags to determine the most suitable tags for the text.
[0080] like Figure 3 The annotation process of the labeling function shown includes four steps: (1) Annotate meaningful words or phrases in the text. (2) Apply heuristic rules to convert word labels into category labels. (3) Perform aggregation methods to determine the most accurate category label. (4) Return the label as the final annotation.
[0081] like Figure 4 The large language model analysis module shown fully supports the annotation process by performing the following tasks: (a) analyzing the samples that the user is currently reviewing and providing references, and (b) examining instances of active learning sampling and analyzing them to obtain heuristic rules, expand the word set, or recommend tag functions.
[0082] like Figure 5 The large language model shown generates hints for three tasks: analysis of annotation results for individual instances, vocabulary expansion, and tag function recommendation. The text highlighted in non-black represents the passed variables, whose values change based on the current annotation state.
[0083] This embodiment also provides a text annotation system driven by large language models, data programming, and active learning, including:
[0084] The labeling task definition module is used to configure the attributes of the current labeling task and obtain the definition of the current labeling task. The attributes include task type, labeling target and label category.
[0085] The tagging function configuration module is used to configure tagging functions according to the definition of the current tagging task. The tagging function tags keywords through a word set, maps the word set to candidate label categories using heuristic rules, and aggregates the candidate label categories to obtain the most suitable label for the text using an aggregation method.
[0086] The active learning sampling module is used to select the sample with the highest uncertainty using an active learning algorithm. The uncertainty is obtained through model uncertainty sampling, label function uncertainty sampling, and unlabeled samples.
[0087] The large language model analysis module is used to analyze selected samples using a large language model, enabling word set expansion, tag function recommendation, and sample analysis.
[0088] like Figures 6-8 The user interface for this system is shown. The system presents six interconnected views:
[0089] (a) Labeling Function Configuration View. This view implements the task definition function, allowing users to specify the core settings of the current labeling task. Users can define the type of labeling task, labeling target, and label category through input boxes and option selections. All labeling functions created in the system will follow this configuration, thus ensuring consistency throughout the labeling process. After defining several labeling functions, users can also click the "Fine-tuning" button in the configuration view to label the current dataset using the current labeling function.
[0090] (b) Tag Function Control View. This view is used to create and manage tag functions, all of which are displayed as an executable list. Tag functions are added to this list after submission. When the user clicks the "Create Tag Function" button, the tag function construction view opens. Users can name the tag function and assign aliases to targets within specific tasks. The left side of the view lists the current vocabulary; users can expand the set to view specific words. By clicking the "+" button, users can combine vocabulary sets and assign them tag categories to form a rule. The generated rule is displayed at the bottom of the view. Finally, the user clicks the "Submit" button to add the newly created tag function to the executable list.
[0091] (c) Instance Distribution View. This view is designed to help users visually assess the current state and interpret the results of active learning. In this view, each node represents a data instance, its color-coded to indicate its currently assigned label, and the relative distance between nodes reflects their similarity. Users can switch between different active learning strategies by selecting the radio buttons above, with the corresponding sampled data points highlighted. All views related to active learning are dynamically updated with each fine-tuning iteration, reflecting the latest annotation results.
[0092] (d) Instance Analysis View. This view provides users with direct access to individual data instances, enabling detailed examination of text content. Users can manually mark important words, identify language patterns or features, and thus inspire new heuristic rules. Simultaneously, this view supports manual verification of individual instances. Users can browse and analyze different instances by clicking the toggle button or selecting nodes in the distribution map.
[0093] (e) Large Language Model Annotation Recommendation View. For complex instances that are difficult to handle using labeling functions and pose a challenge to human annotators, this view leverages the contextual understanding capabilities of large language models to provide users with recommended labels and detailed explanations of their reasoning. The large language model's recommendations help users validate data labels and gain insights to design more effective labeling functions.
[0094] (f) Large Language Model Tag Function Recommendation View. This view leverages the generative capabilities of the large language model to help users construct tag functions, including expanding the vocabulary set and recommending tag functions. The "Vocabulary Set Expansion" function displays words from actively learned samples that match the existing vocabulary set; the "Function Recommendation" function recommends tag functions to users based on heuristic rules extracted from these samples. For ease of use, the system allows users to directly apply the recommended vocabulary set and tag functions with a single click.
[0095] like Figure 9-10 This shows a view of tagging function creation. Users are free to define heuristic rules and create tagging functions. All created rules are displayed below for review and management.
[0096] like Figure 1 The system workflow is illustrated below: First, users initially label the dataset using a small amount of manual word tagging and the creation of a basic labeling function. In this embodiment, the distribution view is updated after each round of labeling, allowing users to explore and select more informative samples using an active learning strategy. Next, the system analyzes the selected samples using a large language model and provides recommendations, which users can use to revise labels, adopt suggested words, or summarize new heuristics. Finally, these improvements are incorporated into a more refined labeling function, achieving continuous iterative optimization of labeling quality. During the iteration process, users can filter and evaluate samples through the active learning view. When a user determines that no more samples with significant information gain have appeared, they can actively choose to end the iterative labeling process.
[0097] Among the different functional views, the Active Learning view visually presents sample differences through a distribution map, effectively guiding users to select samples with higher information content and making complex sampling strategies easier to understand, thereby significantly improving annotation efficiency. The Large Language Analysis view excels in automatic word tagging and handling complex samples, not only simplifying the time-consuming task of word expansion but also helping users quickly handle instances that are difficult to classify or to construct tagging functions for. The Tagging Function view allows users to generate tagging functions in batches with fewer operations, significantly lowering the barrier to coding; however, its configuration logic still has room for improvement in terms of ease of understanding, especially requiring more guidance for new users. Overall, the three views in this embodiment work together to provide users with intuitive, efficient, and flexible annotation support, significantly improving the overall quality and efficiency of data annotation.
[0098] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.
Claims
1. A text annotation method driven by large language models, data programming, and active learning, characterized in that, include: Configure attributes for the current annotation task to obtain the definition of the current annotation task, wherein the attributes include task type, label target, and label category; According to the definition of the current annotation task, a tagging function is configured. The tagging function tags keywords through a word set, maps the word set to candidate tag categories using heuristic rules, and aggregates the candidate tag categories to obtain the most suitable tag for the text using an aggregation method. An active learning algorithm is used to select the sample with the highest uncertainty, wherein the uncertainty is obtained through model uncertainty sampling, label function uncertainty sampling, and unlabeled samples; By analyzing the selected samples using a large language model, we can achieve word set expansion, tag function recommendation, and sample analysis.
2. The text annotation method according to claim 1, characterized in that, The task types include text classification tasks and target-specific annotation tasks; The labeling target is used to specify the labeling target for the specific target labeling task; The tag category is used to define the set of tags available for the task.
3. The text annotation method according to claim 1, characterized in that, The heuristic rules are used for direct mapping of a single word set to a label category, as well as sequential mapping of multiple word sets in combination.
4. The text annotation method according to claim 1, characterized in that, Aggregating the candidate tag categories using an aggregation method includes: The nearest neighbor method is used to determine the position of the target word in the text and search for the nearest tag category; Alternatively, a window analysis method can be used to set a radius range centered on the target, and generate an overall label within a local window through majority voting.
5. The text annotation method according to claim 1, characterized in that, The model uncertainty sampling includes: using data generated by the labeling function to train a pre-trained model to obtain semantic relationships between different data objects, generating a prediction probability for each data, and selecting the sample with the lowest confidence based on the prediction probability; The uncertainty sampling of the labeling function selects samples based on the divergence of multiple labeling functions through a committee query algorithm.
6. The text annotation method according to claim 1, characterized in that, The vocabulary expansion includes: analyzing the selected samples through the large language model and automatically identifying words related to each vocabulary set to achieve vocabulary expansion.
7. A text annotation system driven by a large language model, data programming, and active learning, for implementing the text annotation method as described in any one of claims 1-6, characterized in that, include The labeling task definition module is used to configure the attributes of the current labeling task and obtain the definition of the current labeling task. The attributes include task type, labeling target and label category. The tagging function configuration module is used to configure the tagging function according to the definition of the current tagging task. The tagging function tags keywords through a word set, maps the word set to candidate tag categories using heuristic rules, and aggregates the candidate tag categories to obtain the most suitable tag for the text using an aggregation method. An active learning sampling module is used to select the sample with the highest uncertainty using an active learning algorithm, wherein the uncertainty is obtained through model uncertainty sampling, label function uncertainty sampling, and unlabeled samples; The large language model analysis module is used to analyze selected samples using a large language model, enabling word set expansion, tag function recommendation, and sample analysis.