Big model intelligent label synthesis and data automatic labeling integrated method and system

Abstract

The application provides a large model intelligent label synthesis and data automatic labeling integrated method and system, and belongs to the technical field of label synthesis and data labeling. The method comprises the following steps: performing semantic embedding and robust clustering on text data to obtain a stable cluster set; when new data is introduced, the semantic consistency alignment of cross-round clustering results is realized through a confusion matrix matching strategy to suppress label drift; an editable hierarchical label directed acyclic graph is maintained to support label system evolution; a large language model is driven based on representative samples to generate high-quality and interpretable cluster-level semantic labels; for samples that are not attributed or have low confidence in clustering, automatic labeling and confidence evaluation are performed through context learning on the large model; the cluster-level labels are propagated to instances and combined with the automatic labeling results to construct a full-process traceable label management mechanism. The application improves the efficiency, quality and consistency of text labeling, and provides strong support for large model training and intelligent data management.

Description

Integrated Method and System for Intelligent Tag Synthesis and Automatic Data Annotation of Large Models Technical Field

[0001] This invention relates to the field of tag synthesis and data annotation technology, and in particular to an integrated method and system for intelligent tag synthesis and automatic data annotation of large models. Background Technology

[0002] With the rapid development of artificial intelligence and big data technologies, text data has become a key resource driving the performance of language models and intelligent decision-making. Especially in the fields of natural language processing and deep learning, high-quality labeled data directly determines the generalization ability and semantic stability of the model. However, with the diversification of text sources and the increase in update frequency, traditional methods relying on manual or static annotation are difficult to adapt to the complex scenarios of continuous text data generation, dynamic semantic changes, and multi-round feedback in reality, resulting in low annotation efficiency, high cost, and unstable quality.

[0003] Against this backdrop, large-scale intelligent label synthesis and automatic data annotation technologies have gradually become core directions for improving the efficiency and intelligence level of text data utilization. Label synthesis refers to the process of automatically generating, completing, or fusing labels from existing or partially annotated text using text features, semantic relationships, and model output results; automatic annotation, on the other hand, uses algorithmic models to infer and assign labels to input text, achieving automatic mapping from text to labels. The synergy of these two technologies can reduce the cost of manual annotation while enabling the dynamic evolution of the label system, maintenance of semantic consistency, and automatic optimization.

[0004] Existing technologies have explored this field in various ways. Early studies mostly employed rule-based or clustering-based text label generation methods, generating candidate labels through feature similarity or keyword matching. However, these methods have limited ability to understand complex semantics and contextual relationships. With the development of deep learning and generative models, text label synthesis methods based on pre-trained large models have emerged, capable of capturing high-dimensional semantic structures and achieving more accurate label generation and semantic abstraction. However, most of these methods are designed for static or one-time text datasets and struggle to handle multi-source input, incremental updates, or continuously evolving text data in real-world scenarios, lacking adaptive structure optimization and label evolution mechanisms.

[0005] Meanwhile, current automatic labeling systems typically train models on fixed text datasets and then assign labels to new samples using predicted probabilities or heuristic rules. When text distribution changes or topic structure evolves, the labels output by the model often exhibit semantic drift and labeling instability. Furthermore, the lack of an efficient closed-loop mechanism between automatic labeling results and manual review prevents the system from continuously optimizing label quality and assignment strategies based on user or model feedback, resulting in an inefficient cycle of "automatic generation—manual correction—duplication bias."

[0006] In practical applications, the text data annotation process is itself a dynamic cycle and feedback iteration: new text continuously enters the system, some samples are automatically annotated and then manually reviewed and corrected, and these corrections in turn affect the next round of label generation and model optimization. Without a unified coordination and governance mechanism, the labeling system is prone to problems such as redundancy, conflict, or semantic inconsistency in multiple iterations, resulting in a lack of coherence and interpretability at the global level.

[0007] Therefore, there is an urgent need for a large-model-driven integrated method and system for intelligent tag synthesis and automatic annotation that can operate in a multi-source, dynamic, and continuously feedback text environment. Through structured semantic modeling and a manageable tag evolution mechanism, it can achieve semantically consistent tag generation, dynamic optimization, and automatic allocation, thereby constructing a self-learning and self-evolving intelligent text annotation system that can significantly reduce manual costs and improve data processing efficiency while ensuring annotation quality. Summary of the Invention

[0008] This invention provides an integrated method and system for large-scale intelligent label synthesis and automatic data annotation, which addresses the shortcomings of existing technologies and significantly reduces labor costs and improves data processing efficiency while ensuring annotation quality.

[0009] In a first aspect, this invention provides an integrated method for large-scale intelligent label synthesis and automatic data annotation, comprising: acquiring multi-source text data and performing semantic representation learning through a text encoder to obtain a set of embedded feature vectors for the samples; performing clustering processing on the set of embedded feature vectors and optimizing the clustering results through outlier filtering and cluster size constraints to obtain an optimized cluster set; when receiving new text data or replacing clustering results, performing semantic consistency alignment on the clustering results to achieve semantic alignment of cross-round clustering results; maintaining a directed acyclic graph of hierarchical labels with clusters as nodes, and supporting... The process involves: performing editing operations on nodes; selecting representative samples based on the distance between the sample and the cluster center for each optimized cluster, and using a large language model to generate cluster-level semantic labels based on the representative samples; automatically labeling text samples that do not belong to any cluster or whose attribution confidence score is lower than a preset attribution confidence threshold using the cluster-level semantic labels and representative samples as context, and outputting label confidence scores; propagating the cluster-level semantic labels to text samples within the cluster to form instance-by-instance semantic labels, merging the automatic labeling results, and recording the label source, generation rules, and propagation information.

[0010] According to the integrated method for intelligent label synthesis and automatic data annotation of large models provided by the present invention, the method further includes closed-loop control and reentry operation steps: the controller coordinates the processes of clustering, semantic consistency alignment, label generation, automatic annotation, and label propagation; when a manual review conclusion is received, the confidence score of the automatic annotation output is lower than a preset threshold, or there are samples with multiple label conflicts, a local reentry mechanism is triggered; the local reentry mechanism selectively re-executes the affected clustering, label synthesis, automatic annotation, or propagation operations to achieve continuous optimization of the label system.

[0011] According to the integrated method for large-scale intelligent label synthesis and automatic data annotation provided by the present invention, the optimization of clustering results through outlier sample filtering and cluster size constraints includes: performing initial clustering on the embedded feature vector set using a preset clustering algorithm to obtain initial cluster division; setting a threshold for the minimum cluster size and removing clusters with fewer samples than the threshold; calculating the distance set from non-outlier samples to the cluster center, calculating a distance threshold based on a preset percentile, and marking samples with a distance exceeding the distance threshold as outliers for filtering.

[0012] According to the integrated method for large-scale intelligent label synthesis and automatic data annotation provided by the present invention, the semantic consistency alignment of the clustering results includes: when there are alternative clustering results, constructing a confusion matrix for text samples that are not outliers and are simultaneously assigned values ​​by two clustering partitions; determining the one-to-one correspondence between clusters in different partitions through a matching optimization strategy, and setting aside or re-examining unmatched samples.

[0013] The integrated method for large-scale intelligent label synthesis and automatic data annotation provided by this invention maintains a hierarchical directed acyclic graph of labels using clusters as nodes. This includes: recording the identifier, cluster center vector, representative sample set, and parent-child relationship of each node; when the variance of the distance between a sample within a cluster and the cluster center exceeds a preset splitting threshold, a splitting operation is performed, re-clustering the subsets of the cluster and generating new nodes; during the merging operation, multiple nodes with a cluster center distance less than a preset merging threshold are aggregated into upper-level nodes, and the cluster centers are recalculated; all operations retain auditable records, enabling version management of the label structure.

[0014] According to the integrated method for large-scale intelligent label synthesis and automatic data annotation provided by the present invention, the step of calling a large language model to generate cluster-level semantic labels based on the representative samples includes: inputting the content and context information of the representative samples into the large language model; and processing the data using the large language model to generate cluster-level semantic labels.

[0015] According to the integrated method for large-scale intelligent label synthesis and automatic data annotation provided by the present invention, the automatic annotation of text samples using a large language model and the output of confidence scores include: constructing automatic annotation prompts to infer labels for new samples using cluster-level labels and their representative samples as semantic context; directly generating instance-level labels for samples with confidence scores greater than or equal to a preset threshold; entering a review pool and marking the reasons for samples with automatic annotation output confidence scores lower than a preset threshold or with multiple label conflicts; and recording the decision basis and output confidence scores for automatic annotation.

[0016] According to the integrated method for large-scale intelligent label synthesis and automatic data annotation provided by the present invention, the step of propagating cluster-level semantic labels to text samples within the cluster to form instance-by-instance semantic labels, merging automatic annotation results, and recording label sources, generation rules, and propagation information includes: assigning cluster-level semantic labels to text samples within the cluster through a batch propagation mechanism to form instance-by-instance semantic labels; merging and resolving conflicts between the label results generated by automatic annotation and the semantic labels formed by propagation, and uniformly writing them into the label library; and recording label sources, generation rules, model parameters, and execution version information during the propagation and merging process.

[0017] According to the integrated method for intelligent label synthesis and automatic data annotation of large models provided by the present invention, the decision basis for automatic annotation includes: the prompting context on which automatic annotation is based, the version identifier of the large language model used, and the confidence threshold.

[0018] Secondly, the present invention also provides an integrated system for large-scale intelligent label synthesis and automatic data annotation, comprising:

[0019] The data access and feature embedding module is used to acquire multi-source text data and learn semantic representation through a text encoder to obtain a set of embedded feature vectors for the samples.

[0020] The robust clustering and outlier gating module is used to perform clustering processing on the embedded feature vector set, and optimize the clustering results by outlier sample filtering and cluster size constraints to obtain an optimized cluster set.

[0021] The semantic consistency alignment module is used to perform semantic consistency alignment on the clustering results when new text data or alternative clustering results are received, so as to achieve semantic alignment of clustering results across rounds.

[0022] The hierarchical label structure maintenance module is used to maintain a directed acyclic graph of hierarchical labels with clusters as nodes, and supports editing operations on the nodes;

[0023] The representative sample and label synthesis module is used to select representative samples for each optimized cluster based on the distance between the sample and the cluster center, and call the large language model to generate cluster-level semantic labels based on the representative samples;

[0024] The automatic labeling module is used to automatically label text samples that do not belong to any cluster or whose attribution confidence score is lower than a preset attribution confidence threshold, using the cluster-level semantic labels and representative samples as context, and output the label confidence score.

[0025] The tag propagation and governance module is used to propagate cluster-level semantic tags to text samples within the cluster, forming instance-by-instance semantic tags, merging automatic annotation results, and recording tag sources, generation rules, and propagation information;

[0026] The control feedback module is used to coordinate the various modules and trigger the local reentry mechanism.

[0027] The integrated method and system for intelligent label synthesis and automatic data annotation of large models provided by this invention establishes a governable, traceable, and scalable text label synthesis and automatic annotation system by integrating cluster structure analysis, semantic label generation, and closed-loop self-learning mechanism. This provides a high-quality, stable, and consistent annotation foundation for intelligent text data management and large model training, and has the following advantages compared to existing technologies:

[0028] (1) Stable tag generation and semantic consistency: Robust clustering and percentile gating reduce the impact of text noise, and combined with consistency alignment mechanism effectively suppress tag drift, ensuring semantic consistency under multi-round operation and continuous feedback conditions;

[0029] (2) The label system is governable and traceable: Through hierarchical label structure management and operation log recording, the entire life cycle of text labels from generation and dissemination to revision is traceable and supervised;

[0030] (3) Intelligent semantic tag synthesis: With the help of the semantic understanding and generation capabilities of the large language model, the representative text samples are aggregated and analyzed and tags are generated to achieve high-quality, interpretable and reusable semantic tag synthesis;

[0031] (4) Closed-loop self-optimization mechanism: The controller constructs a reentrant loop structure and automatically performs local recalculation when receiving manual review or external feedback, so as to realize system-level self-correction and continuous optimization;

[0032] (5) High efficiency and low cost: Through automated annotation and intelligent tag synthesis, manual intervention is greatly reduced, the efficiency and accuracy of text annotation are improved, and the overall annotation cost is significantly reduced;

[0033] (6) Wide range of applications and strong scalability: It is suitable for multi-source, multi-domain and continuously updated text data environments, can work in conjunction with manual review processes, and supports dynamic tag evolution and intelligent data governance applications. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0035] Figure 1 is one of the flowcharts of the integrated method for large-scale intelligent label synthesis and automatic data annotation provided by the present invention;

[0036] Figure 2 is a second flowchart of the integrated method for large-scale intelligent label synthesis and automatic data annotation provided by the present invention;

[0037] Figure 3 is a schematic diagram of the integrated system for large-scale intelligent label synthesis and automatic data annotation provided by the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0039] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0040] The integrated method and system for large-scale intelligent label synthesis and automatic data annotation provided by the embodiments of the present invention are described below with reference to Figures 1-3.

[0041] Figure 1 is one of the flowcharts illustrating the integrated method for large-scale intelligent label synthesis and automatic data annotation provided by the present invention, and Figure 2 is another flowchart illustrating the integrated method for large-scale intelligent label synthesis and automatic data annotation provided by the present invention. The technical solution of the present invention will be described below with reference to Figures 1 and 2. The present invention includes, but is not limited to, the following steps:

[0042] Step 101: Obtain multi-source text data and perform semantic representation learning through a text encoder to obtain a set of embedded feature vectors for the samples.

[0043] This step involves data acquisition and semantic representation. It involves acquiring multi-source text data to be processed (including various types of text such as user comments, business logs, question-and-answer dialogues, and web page content), and then using a text encoder to learn semantic representations to obtain a set of embedded feature vectors for the samples, which are used for subsequent clustering and label synthesis.

[0044] Step 102: Perform clustering processing on the embedded feature vector set, and optimize the clustering results by filtering outlier samples and constraining cluster size to obtain an optimized cluster set.

[0045] This step is a robust structure induction step. Optionally, a basic clustering algorithm (such as the K-means algorithm) can be run in the semantic feature space to optimize the initial clustering results. By setting a minimum cluster size threshold to remove small cluster samples and using a percentile outlier gating mechanism to filter the distance from the sample to the cluster center, outlier samples and long-tail noise can be effectively eliminated, resulting in structurally stable, semantically concentrated, and highly representative clusters and cluster centers.

[0046] Based on the above filtering operations, the present invention can reduce the interference of long-tail samples on clustering results and improve the stability and semantic representativeness of cluster centers.

[0047] Step 103: When new text data or alternative clustering results are received, perform semantic consistency alignment on the clustering results to achieve semantic alignment of clustering results across rounds.

[0048] This step is a semantic consistency alignment step. When new text data or alternative clustering results are received, a clustering confusion matrix is ​​constructed for samples that are not outliers and are assigned values ​​by both partitions. The one-to-one correspondence between clusters in different partitions is determined by a matching optimization strategy to achieve semantic alignment of clustering results across rounds and ensure the consistency and comparability of the labeling system in multiple rounds of operation.

[0049] Step 104: Maintain a directed acyclic graph with hierarchical labels using clusters as nodes, and support editing operations on nodes.

[0050] This step involves maintaining the hierarchical label structure. Using clusters as nodes, it maintains an editable directed acyclic graph (DAG) of hierarchical labels, supporting operations such as splitting, merging, migrating, or deleting nodes. The system logs each editing operation during execution, achieving version management, traceability, and semantic governance of the label structure.

[0051] As an optional embodiment, this invention maintains a directed acyclic graph of hierarchical labels using clusters as nodes. This includes recording the identifier, cluster center vector, representative text sample set, and parent-child relationships of each cluster as a node. It supports editing operations on nodes such as splitting (re-clustering subsets and attaching them as child nodes), merging (aggregating multiple semantically similar nodes into a higher-level node and recalculating the cluster center), migrating (updating the cluster center and refreshing the representative samples), and deleting (removing subtrees and marking members as outliers). All operations retain auditable records to achieve version management and traceability of the label structure.

[0052] Step 105: For each optimized cluster, select a representative sample based on the distance between the sample and the cluster center, and call the large language model to generate cluster-level semantic labels based on the representative sample.

[0053] This step is a representative sample-driven label synthesis step. For each optimized cluster, several representative samples are selected according to their distance from the cluster center, from closest to furthest. The system calls the Large Language Model (LLM) to perform semantic understanding, topic extraction, and concept aggregation on these representative samples, generating concise, unified, and interpretable cluster-level semantic labels. During the label generation process, automatic adjustment and optimization can be achieved by combining contextual information, historical labels, and naming rules.

[0054] As an optional embodiment, the present invention selects representative samples based on the distance between the sample and the cluster center, and calls a large language model to generate cluster-level semantic labels based on the representative samples, including:

[0055] Based on the distance between the text samples and the cluster center, sort them from smallest to largest, select the closest samples as a representative sample set to cover the main semantic features within the cluster and avoid interference from tail samples; provide the content and context information of the representative text samples to the Large Language Model (LLM) to generate cluster-level semantic labels with consistency, interpretability and reusability, and govern and unify them according to the preset naming conventions or label dictionary when needed.

[0056] Step 106: For text samples that do not belong to any cluster or whose attribution confidence score is lower than the preset attribution confidence threshold, use the cluster-level semantic labels and representative samples as context to automatically label the text samples using a large language model and output the label confidence score.

[0057] It should be noted that the attribution confidence score here is the result of the above clustering process. Essentially, it is a quantitative measure of the certainty or goodness of fit of each sample to its cluster by the clustering algorithm. This invention not only automatically labels text samples that do not belong to any cluster in the above clustering process, but also automatically labels text samples that have been assigned to a cluster but whose attribution confidence is lower than the preset attribution confidence threshold.

[0058] This step is an automatic annotation step based on a large model. After obtaining cluster-level semantic labels, the system automatically annotates text samples that are unassigned or have low assignment confidence based on the large model. The steps include:

[0059] a) Using cluster-level labels and their representative samples as semantic context, construct automatic labeling prompts or classification heads to perform label inference and confidence assessment on new samples;

[0060] b) Generate instance-level labels directly for high-confidence samples; for low-confidence or conflicting samples, enter the verification pool and mark the reasons (such as insufficient confidence interval, conflict of nearby clusters, abnormal patterns, etc.).

[0061] c) Record the decision-making basis (context used, model version, threshold) and output confidence score automatically to facilitate subsequent governance and backtracking.

[0062] Step 107: Propagate the cluster-level semantic tags to the text samples within the cluster to form instance-by-instance semantic tags, merge the automatic annotation results, and record the tag source, generation rules, and propagation information.

[0063] This step involves controlled propagation and label governance. The invention assigns the generated cluster-level semantic labels to text samples within a cluster through a batch propagation mechanism, forming refined labels for each instance and creating a maintainable label library. Furthermore, after the large model automatically labels unassigned or low-confidence text samples, the invention can merge the automatic labeling results and uniformly write them into the label library to update the label library.

[0064] During the dissemination and merging process, this invention records information such as the source of the label, generation rules, model parameters, and execution version, and constructs a fully traceable label governance mechanism to ensure the transparency, consistency, and regulatory oversight of the label evolution process.

[0065] As an optional embodiment, this invention propagates cluster-level semantic tags to text samples of each cluster member based on hierarchical clustering graphs and consistency alignment results, forming refined text tags for each instance; and merges and resolves conflicts in the results of automatic annotation in step 106; the system records the source (representative sample, model version, parameter configuration), timestamp, propagation rules and scope of influence in each tag generation, propagation and editing process, and supports tag rollback and difference comparison when necessary, thereby achieving full-process traceability and governance.

[0066] Based on the above embodiments, as an optional embodiment, the present invention further includes: closed-loop control and re-entry operation steps (not shown in Figure 1, see Figure 2): The present invention coordinates and arranges the above steps through a central controller, forming a three-stage cyclic mode of "bootstrap - refinement - label-only". When a manual review conclusion or external feedback signal is detected, or when automatic labeling produces low confidence (below a preset threshold) / label conflict, the controller can trigger a local re-entry mechanism (Re-entrant Mode) to automatically and selectively re-execute the affected clustering, synthesis, automatic labeling, or propagation modules, thereby achieving continuous self-optimization and dynamic evolution of the label system and model, avoiding global recalculation, and improving feedback response efficiency and labeling stability.

[0067] Based on the above embodiments, as an optional implementation, this invention can simultaneously maintain two types of text-by-text instance tags: one is working tags (original cluster tags generated by the clustering structure), and the other is refined tags (semantic tags synthesized by a large language model, automatically labeled, and propagated). Both types of tags maintain consistency constraints throughout the entire process and support difference analysis and version backtracking to ensure the continuity and interpretability of tag semantics.

[0068] Based on the above embodiments, as an optional embodiment, both the encoder and the clusterer are replaceable components, and the distance metric can be cosine distance, Euclidean distance, or Manhattan distance; parameters such as minimum cluster size threshold, outlier percentile threshold, number of representative samples, and automatic labeling confidence threshold can be configured according to the size of the text corpus and domain characteristics to adapt to the running requirements of different text datasets.

[0069] Based on the above embodiments, as an optional embodiment, the data type is limited to text data. This invention is designed for unstructured or semi-structured text scenarios, can operate under conditions of multi-source input, batch incremental and multi-round feedback, and can be linked with manual review or active learning strategies to form an efficient and manageable human-machine collaborative annotation closed loop.

[0070] To more clearly explain the technical solution of the present invention and to facilitate understanding of the advantages of the present invention compared with the prior art, the technical solution of the present invention will be described below in conjunction with a complete system-side embodiment.

[0071] In this embodiment, an unstructured dataset containing multi-source text samples is used as an example to demonstrate the operation of the system described in this invention.

[0072] Figure 3 is a schematic diagram of the integrated system for large-scale intelligent label synthesis and automatic data annotation provided by the present invention. As shown in Figure 3, the system includes: a data access and feature embedding module 301, a robust clustering and outlier gating module 302, a semantic consistency alignment module 303, a hierarchical label structure maintenance module 304, a representative sample and label synthesis module 305, an automatic annotation module 306, a label propagation and governance module 307, and a control feedback module 308.

[0073] (1) Data access and feature embedding module, used to acquire multi-source text data and learn semantic representation through text encoder to obtain the set of embedded feature vectors of samples.

[0074] The system first accesses unlabeled text datasets from multiple business sources (such as user feedback, social comments, customer service dialogue logs, Q&A platform data, etc.).

[0075] For these unstructured texts, the system calls a pre-trained language model (such as BERT, RoBERTa, or its own large model encoder) to extract the text semantic embeddings, resulting in a sample set:

[0076]

[0077] in, For vector dimensions, The total number of samples, For the sample set, This is the i-th sample. The resulting set of embedding vectors is input into the subsequent clustering module as the basis of the representation space, providing a measurable semantic representation for the formation of the label structure.

[0078] (2) Robust clustering and outlier gating module, used to perform clustering processing on the embedded feature vector set, and optimize the clustering results by outlier sample filtering and cluster size constraints to obtain the optimized cluster set.

[0079] Optionally, an initial clustering algorithm is used to perform initial clustering on the set of embedded feature vectors to obtain an initial cluster division; a threshold for the minimum cluster size is set, and clusters with fewer than the threshold number of samples are removed; the distance set from non-outliers to the cluster center is calculated, a distance threshold is calculated based on a preset percentile, and samples with a distance exceeding the distance threshold are marked as outliers and filtered out.

[0080] Specifically, the system embeds a set of vectors. Execute basic clustering algorithms (such as K-Means or DBSCAN) to obtain initial cluster partitions. To eliminate the influence of small clusters and noisy samples, the system sets a minimum cluster size threshold. The number of samples is less than The clusters are labeled as outlier clusters.

[0081] Subsequently, the system calculates the set of distances from each sample to its cluster center. And estimate the distance distribution of the th Percentile distance threshold t.

[0082] for The system marks the samples as outliers and removes them, thus obtaining robust cluster partitioning results. With stable cluster center set This step significantly reduces the interference of tail distribution on clustering results, improving the representativeness and consistency of clusters.

[0083] (3) Semantic consistency alignment module, which is used to perform semantic consistency alignment on the clustering results when new text data or replacement clustering results are received, so as to achieve semantic alignment of clustering results across rounds.

[0084] Optionally, the semantic consistency alignment of the clustering results includes: when alternative clustering results exist, constructing a confusion matrix for text samples that are not outliers and are simultaneously assigned values ​​by both clustering partitions; determining the one-to-one correspondence between clusters in different partitions through a matching optimization strategy, and setting aside or re-examining unmatched samples. Specifically, the execution steps of this module are as follows:

[0085] a) Constructing a co-occurrence confusion matrix: Let the set of historical clusters be... The current new cluster set is The system calculates the intersection of the two on shared samples and constructs a confusion matrix. ;

[0086] b) Global optimal matching: To maximize the inheritance of labels, apply linear sum and distribution algorithms (such as the Hungarian algorithm) to the matrix. The solution is to find a mapping relationship that maximizes the total number of co-occurring samples of all matching cluster pairs, thereby determining the best correspondence path between the old and new clusters;

[0087] c) Threshold determination and attribute inheritance: Calculate the overlap rate of the matching pair. If it exceeds the preset alignment threshold, the semantics are determined to be stable and the new cluster directly inherits the semantic tag ID and name of the old cluster without re-examination. If it is below the threshold, it is marked as "semantic drift" or "new cluster" and the subsequent modules are triggered to re-synthesize the tags.

[0088] (4) Hierarchical label structure maintenance module, which is used to maintain a directed acyclic graph of hierarchical labels with clusters as nodes, and supports editing operations on nodes.

[0089] The system constructs a hierarchical cluster graph with each cluster as a node. Each node contains attributes: node ID, cluster center vector, representative sample set, parent-child relationship, and status flag. The system supports four types of editable operations: Split, Merge, Relocate, and Delete.

[0090] For example, when the semantic span within a cluster is large, the system can invoke a split operation to re-cluster the subsets, generating two new nodes and recording their parent node relationships in the graph structure. When two clusters are identified as semantically similar by LLM, the system can perform a merge operation to form a new upper-level node. All editing operations are written to the operation log, achieving version management and traceability maintenance of the tag system.

[0091] The semantic span within a cluster can be measured by the distance variance between samples within the cluster and the cluster center. Specifically, when the distance variance between samples within the cluster and the cluster center exceeds a preset splitting threshold, a splitting operation is performed to re-cluster the subsets of the cluster and generate new nodes.

[0092] Among them, semantic similarity can be identified by cluster center distance. When performing the merging operation, multiple nodes with cluster center distance less than the preset merging threshold are aggregated into upper-level nodes and the cluster center is recalculated.

[0093] (5) Representative sample and label synthesis module, used to select representative samples for each optimized cluster based on the distance between the sample and the cluster center, and call the large language model to generate cluster-level semantic labels based on the representative samples.

[0094] For each stable cluster , To stabilize the number of clusters, the system calculates the distance between the sample and the cluster center. Choose the nearest one One sample is used as a representative sample set. These samples, along with cluster core semantic summaries and historical context, are fed into the Large Language Model (LLM). The model generates concise, interpretable cluster-level labels based on the sample content. .

[0095] For example, in an e-commerce scenario, if the representative samples of a cluster are "slow delivery," "logistics updates are delayed," and "delivery has not arrived," then the semantic label generated by LLM might be "logistics delay problem." This label is then marked as a cluster by the system. The semantic representation of the tag is then used to enter the propagation process.

[0096] (6) Automatic labeling module, used to automatically label text samples that do not belong to any cluster or whose attribution confidence score is lower than the preset attribution confidence threshold, using the cluster-level semantic labels and representative samples as context, and output label confidence.

[0097] Optionally, using cluster-level labels and their representative samples as semantic context, automatic annotation prompts are constructed to infer labels for new samples; instance-level labels are directly generated for samples with confidence scores greater than or equal to a preset threshold; samples with confidence scores lower than a preset threshold or with multiple label conflicts are entered into a review pool and the reasons are marked; the decision basis and output confidence scores of automatic annotation are recorded.

[0098] Specifically, it includes the following steps:

[0099] 1) The system first constructs a semantic context prompt based on the cluster centers and their representative samples, and then calls the Large Language Model (LLM) to perform semantic matching and label inference:

[0100]

[0101] in, Given the input sample, emb is the embedding function for the semantic vector. Let l be the generated set of cluster-level labels, l be the label variable of a single cluster, and P be the probability function of the sample and the label. This represents the predicted label of the current input sample.

[0102] 2) The model simultaneously outputs the confidence score for each sample. The system is based on the confidence threshold. Determine the sample labeling strategy:

[0103] when In this case, the predicted labels are directly used as the automatic labeling results;

[0104] when If a multi-label conflict occurs, the sample will be marked as "pending verification" and the triggering reason will be recorded (such as low confidence, semantic conflict or contextual ambiguity).

[0105] 3) The system writes all automatic annotation results and confidence information into the label governance log, recording the corresponding large model version, prompt template and timestamp to ensure traceability and comparability.

[0106] This module enables the system to quickly label new samples without human intervention, significantly improving the coverage and timeliness of label generation, and providing high-quality preliminary labeling references for subsequent manual review and feedback mechanisms.

[0107] (7) Tag propagation and governance module, which is used to propagate cluster-level semantic tags to text samples within the cluster, form instance-by-instance semantic tags, merge automatic annotation results, and record tag source, generation rules and propagation information.

[0108] For each cluster label The system then distributes these samples in batches to all member samples within the cluster, forming instance-by-instance semantic labels. This module will also automatically merging and resolving conflicts between the generated labels and the semantic labels formed during propagation, and write them into the label library in a unified manner; during the propagation and merging process, it will record the label source, generation rules, model parameters and execution version information, etc., to support subsequent backtracking, auditing and correction.

[0109] (8) Control feedback module, used to coordinate the modules and trigger the local reentry mechanism.

[0110] The system controller is responsible for coordinating the above modules and maintaining the three-stage cycle of "bootstrapping - refinement - automatic annotation". In the bootstrapping stage, the system starts from a zero-label state and establishes an initial label system through LLM label generation; in the refinement stage, the label structure is re-aligned based on human feedback and new data; in the automatic annotation stage, new samples are quickly and automatically labeled and propagated using the existing label system and model context.

[0111] The controller automatically switches modes based on task strategies, feedback signals, or model confidence levels. When it detects that the confidence level of manually reviewed conclusions, external feedback, or automatically labeled outputs is lower than a preset threshold, or when there are conflicts in automatically labeled tags, it triggers a local re-entry mechanism to re-execute the affected clustering, label synthesis, or propagation steps, achieving system-level self-correction and dynamic evolution. Through this mechanism, the system can maintain the consistency, stability, and high quality of semantic labels in dynamic data environments.

[0112] In real-world business testing, this invention was tested on a set of customer service data containing 100,000 unstructured text samples.

[0113] The experimental results are as follows:

[0114] 1) Label consistency improved by approximately 27%, and the accuracy of automatic labeling increased from 78% to 91%;

[0115] 2) The workload of manual labeling is reduced by approximately 65%;

[0116] 3) The maintenance time for the labeling system has been reduced by half;

[0117] 4) In multiple iterations, the stability of semantic tags is significantly enhanced, and partial updates can be performed without full recalculation.

[0118] The results show that the system of the present invention can operate efficiently in a multi-source, dynamic, and continuously feedback data environment, realize an integrated closed loop of label synthesis and automatic annotation, and significantly improve the quality of data assets and the model training effect.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for integrating intelligent label synthesis and automatic data annotation for large-scale models, characterized in that, include: Multi-source text data is acquired, and semantic representation learning is performed through a text encoder to obtain a set of embedded feature vectors for the samples; the set of embedded feature vectors is clustered, and the clustering results are optimized through outlier filtering and cluster size constraints to obtain an optimized cluster set; When new text data or alternative clustering results are received, semantic consistency alignment is performed on the clustering results to achieve semantic alignment of clustering results across rounds. A hierarchical directed acyclic graph of labels is maintained using clusters as nodes, and editing operations on nodes are supported. For each optimized cluster, a representative sample is selected based on the distance between the sample and the cluster center, and a large language model is invoked to generate cluster-level semantic labels based on the representative sample. For text samples that do not belong to any cluster or whose attribution confidence score is lower than a preset attribution confidence threshold, the large language model is used to automatically label the text samples using the cluster-level semantic labels and the representative sample as context, and the label confidence score is output. The cluster-level semantic labels are propagated to text samples within the cluster to form instance-by-instance semantic labels. The automatic labeling results are merged, and the label source, generation rules, and propagation information are recorded.

2. The integrated method for large-scale intelligent label synthesis and automatic data annotation according to claim 1, characterized in that, The method also includes closed-loop control and reentry operation steps: the controller coordinates the processes of clustering, semantic consistency alignment, label generation, automatic annotation, and label propagation; when a manual review conclusion is received, the confidence score of the automatic annotation output is lower than a preset threshold, or there is a conflict in the automatic annotation labels, a local reentry mechanism is triggered; the local reentry mechanism selectively re-executes the affected clustering, label synthesis, automatic annotation, or propagation operations to achieve continuous optimization of the label system.

3. The integrated method for large-scale intelligent label synthesis and automatic data annotation according to claim 1, characterized in that, The optimization of clustering results through outlier filtering and cluster size constraints includes: performing initial clustering on the embedded feature vector set using a preset clustering algorithm to obtain initial cluster division; setting a minimum cluster size threshold to remove clusters with fewer samples than the threshold; calculating the distance set from non-outlier samples to the cluster center, calculating a distance threshold based on preset percentiles, and marking samples with a distance exceeding the distance threshold as outliers for filtering.

4. The integrated method for large-scale intelligent label synthesis and automatic data annotation according to claim 1, characterized in that, The semantic consistency alignment of the clustering results includes: when there are alternative clustering results, constructing a confusion matrix for text samples that are not outliers and are assigned values ​​by both clustering partitions; determining the one-to-one correspondence between clusters in different partitions through a matching optimization strategy, and setting aside or re-examining unmatched samples.

5. The integrated method for large-scale intelligent label synthesis and automatic data annotation according to claim 1, characterized in that, A hierarchical directed acyclic graph of labels is maintained using clusters as nodes. This includes recording the identifier, cluster center vector, representative sample set, and parent-child relationship of each node. When the variance of the distance between a sample within a cluster and the cluster center exceeds a preset splitting threshold, a splitting operation is performed, and the subset of the cluster is re-clustered to generate new nodes. When performing a merging operation, multiple nodes whose cluster center distance is less than a preset merging threshold are aggregated into upper-level nodes, and the cluster centers are recalculated. All operations are recorded in an auditable manner to achieve version management of the label structure.

6. The integrated method for large-scale intelligent label synthesis and automatic data annotation according to claim 1, characterized in that, The step of calling the large language model to generate cluster-level semantic labels based on the representative samples includes: inputting the content and context information of the representative samples into the large language model; and processing the data using the large language model to generate cluster-level semantic labels.

7. The integrated method for large-scale intelligent label synthesis and automatic data annotation according to claim 1, characterized in that, The automatic annotation of text samples using a large language model and the output of confidence scores include: constructing automatic annotation prompts to infer labels for new samples using cluster-level labels and their representative samples as semantic context; directly generating instance-level labels for samples with confidence scores greater than or equal to a preset threshold; entering a review pool and marking the reasons for samples with automatic annotation output confidence scores lower than a preset threshold or with multiple label conflicts; and recording the decision basis and output confidence scores for automatic annotation.

8. The integrated method for large-scale intelligent label synthesis and automatic data annotation according to claim 1, characterized in that, The process of propagating cluster-level semantic tags to text samples within a cluster to form instance-by-instance semantic tags, merging automatic annotation results, and recording tag sources, generation rules, and propagation information includes: assigning cluster-level semantic tags to text samples within a cluster through a batch propagation mechanism to form instance-by-instance semantic tags; merging and resolving conflicts between the automatically annotated tag results and the propagated semantic tags, and uniformly writing them into a tag library; and recording tag sources, generation rules, model parameters, and execution version information during the propagation and merging process.

9. The integrated method for large-scale intelligent label synthesis and automatic data annotation according to claim 7, characterized in that, The decision-making criteria for automatic record annotation include: the context of the prompts on which the automatic record annotation is based, the version identifier of the large language model used, and the confidence threshold.

10. A large-scale intelligent label synthesis and automatic data annotation integrated system, characterized in that, The method for implementing any one of claims 1 to 9 comprises: a data access and feature embedding module for acquiring multi-source text data and performing semantic representation learning through a text encoder to obtain a set of embedded feature vectors for samples; a robust clustering and outlier gating module for performing clustering processing on the set of embedded feature vectors and optimizing the clustering results through outlier sample filtering and cluster size constraints to obtain an optimized cluster set; a semantic consistency alignment module for performing semantic consistency alignment on the clustering results when new text data or alternative clustering results are received, so as to achieve semantic alignment of clustering results across rounds; and a hierarchical label structure maintenance module for maintaining a directed acyclic graph of hierarchical labels with clusters as nodes and supporting editing operations on nodes; The module for synthesizing representative samples and labels is used to select representative samples for each optimized cluster based on the distance between the sample and the cluster center, and to call a large language model to generate cluster-level semantic labels based on the representative samples. The automatic labeling module is used to automatically label text samples that do not belong to any cluster or whose attribution confidence score is lower than a preset attribution confidence threshold, using the cluster-level semantic labels and representative samples as context, and output the label confidence score. The label propagation and governance module is used to propagate the cluster-level semantic labels to text samples within the cluster, forming instance-by-instance semantic labels, merging the automatic labeling results, and recording the label source, generation rules, and propagation information. The control feedback module is used to coordinate the various modules and trigger a local re-entry mechanism.

Images