Automatic data annotation system and method based on large model
Through the large model-based automated data labeling system, the problem of time-consuming and inaccurate traditional data labeling is solved, fast and accurate data labeling is achieved, and labeling efficiency and accuracy are improved.
Patent Information
- Application Number
- CN202510941379.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-09
- Publication Date
- 2025-10-24
AI Technical Summary
Traditional data labeling methods are time-consuming and prone to human errors, resulting in inaccurate labeling results and low efficiency. In addition, existing labeling models are prone to inaccurate labeling.
An automated data labeling system based on a large model is adopted, including a data splitting module, a multi-model management module, a dynamic allocation module, a conflict arbitration module and an efficient labeling model. Through data splitting, multi-model collaborative labeling, dynamic task allocation and rational processing of conflict results, the labeling accuracy and efficiency are improved.
It achieves fast and accurate annotation of complex data sets, saves manpower and material resources, improves annotation speed and accuracy, and reduces the incidence of human errors.
Smart Images
Figure CN120833533A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of data labeling, and particularly relates to an automatic data labeling system and method based on a large model. BACKGROUND
[0002] In many fields such as computer vision, image processing and machine learning, data labeling is an important task. Data labeling refers to assigning corresponding labels or annotations to samples in a data set, so as to perform model training, algorithm evaluation or application development and the like. However, the traditional data labeling method usually needs human participation, and the process is time-consuming and prone to human errors; moreover, there are problems such as inaccurate labeling results and low labeling efficiency.
[0003] Moreover, the current digital labeling usually adopts a set of labeling model, which is prone to cause inaccurate labeling, therefore, the application discloses an automatic data labeling system and method based on a large model to solve the above problems. SUMMARY
[0004] The application aims to overcome the deficiencies of the prior art and provide an automatic data labeling system and method based on a large model, which has accurate labeling and fast processing.
[0005] The technical scheme of the application is as follows:
[0006] An automatic data labeling system and method based on a large model, comprising:
[0007] A data splitting module: used for classifying and splitting an original complex data set into a plurality of sub-data sets according to a preset rule, for subsequent labeling processing by a multi-model management module, the splitting rule including at least one of data type, semantic complexity and labeling task type;
[0008] A multi-model management module: containing a plurality of heterogeneous large models, each large model being trained for different data features or labeling tasks, the training mode including model architecture, training data distribution and labeling granularity, and each large model being capable of performing a labeling task for different sub-data sets;
[0009] A dynamic allocation module: generating an allocation strategy according to the feature matching degree of the sub-data set and the performance of each heterogeneous large model, and allocating the labeling tasks of the plurality of large models, the matching degree being quantified by the labeling accuracy, processing speed and field adaptability of the large model on a validation set;
[0010] A conflict arbitration module: performing confidence weighting or semantic fusion processing on the conflict results generated by different models labeling the same data;
[0011] An efficient labeling model is used to establish a new large model, the new large model integrates historical labeling records of multiple large models, and the completion speed of different types of labeling tasks is improved by reducing the accuracy;
[0012] A version management module records the operation log of the labeling process and retains historical records and historical versions.
[0013] Further, the data splitting module further comprises:
[0014] A primary splitting unit performs coarse-grained division according to data sources or formats;
[0015] A deep splitting unit merges semantically similar data into the same subset through a clustering algorithm.
[0016] Further, the large model in the multi-model management module comprises at least two combinations of:
[0017] A visual large model: a ViT architecture optimized for image segmentation tasks;
[0018] A text large model: a sparse attention mechanism Transformer adapted for long text sequence labeling;
[0019] A multi-modal large model: a joint embedding space model trained using cross-modal contrastive learning.
[0020] Further, the dynamic allocation module further comprises:
[0021] A data monitoring allocation module: real-time monitoring of the calculation resource occupancy rate of each model, preferentially allocating high-complexity data to idle models;
[0022] An inefficient large model retraining module: dynamically adjusting the allocation weight according to historical labeling records, and triggering a retraining mechanism for continuously inefficient models.
[0023] Further, the conflict arbitration module further comprises:
[0024] An automatic correction module: used to establish a domain knowledge graph verification channel, to verify the semantic rationality of the conflict labeling results, and to automatically correct unreasonable annotations;
[0025] An expert module: used to set up an expert voting mechanism, when the degree of disagreement of automatic arbitration exceeds a threshold, triggering online collaborative labeling by domain experts.
[0026] Compared with the prior art, the beneficial effects of the present application are:
[0027] The system can quickly split and classify complex data and quickly label each type of sub-data set through the cooperation of the data splitting module, the multi-model management module, the dynamic allocation module and the efficient labeling model, and can reasonably process the conflict results of the same labeling through the conflict arbitration module, thereby improving the accuracy of labeling.
[0028] An automatic data labeling method based on a large model, comprising the following steps:
[0029] Step S1: dynamic data splitting:
[0030] The primary splitting unit performs coarse-grained division according to data sources / formats, and the depth splitting unit merges data with similar semantics into the same subset using a clustering algorithm to form a multi-dimensional classification label system;
[0031] Step S2: multi-model collaborative labeling:
[0032] The heterogeneous large model in the multi-model management module is called, the optimal model combination is automatically matched according to the characteristics of the sub-data set, and the labeling task is executed in parallel;
[0033] Step S3: intelligent task allocation:
[0034] The dynamic allocation module calculates the performance indicators of each model on the validation set in real time, dynamically adjusts the task queue in combination with the current GPU utilization, and enables the load balancing strategy for high-complexity data;
[0035] Step S4: conflict semantic fusion:
[0036] When the multi-model labeling results of the same data conflict, a confidence weighted voting mechanism or a semantic consistency verification based on a knowledge graph is used to output the fused labeling results;
[0037] Step S5: incremental model optimization:
[0038] The historical labeling records are used as training data, and the large model is fine-tuned through the efficient labeling model using a curriculum learning strategy, which improves the labeling speed by 20%-30% while maintaining an accuracy of more than 90%;
[0039] Step S6: full-process version tracing:
[0040] The model version, parameter configuration and operation instruction in the labeling process are recorded, the labeling results of any version can be traced back according to the timestamp, and the difference comparison accuracy reaches 0.01%.
[0041] Compared with the prior art, the method has the advantages that:
[0042] The method can accurately label and quickly process complex data sets, saving manpower and resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 It is a schematic diagram of the module structure of the present invention. DETAILED DESCRIPTION
[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0045] like Figure 1 As shown, an automatic data annotation system based on a large model includes:
[0046] Data splitting module: used to classify and split the original complex data set into multiple sub-datasets according to preset rules, which are used for annotation processing by the subsequent multi-model management module. The splitting rules include at least one of data type, semantic complexity, and annotation task type;
[0047] Data type: data of different modalities such as images, text, and audio.
[0048] Semantic complexity: divided according to the difficulty of understanding the data (such as simple object recognition vs. complex scene understanding).
[0049] Labeling task type: classification, segmentation, entity recognition and other different task requirements;
[0050] Multi-model management module: Contains multiple heterogeneous large models, each trained for different data features or labeling tasks. The training mode includes model architecture, training data distribution, and labeling granularity. Each heterogeneous large model can perform labeling tasks for different sub-datasets.
[0051] Model architecture: For example, ViT (Visual Transformer) is suitable for images, and BERT is suitable for text.
[0052] Training data distribution: Fine-tune the model for different data distributions (e.g., medical images vs. natural scenes).
[0053] Annotation granularity: e.g., optimizing the difference between pixel-level segmentation and whole-image classification;
[0054] ViT model annotates crack areas in photos (pixel-level segmentation)
[0055] The text model extracts the "hazard warning" keyword from the instruction manual (entity recognition)
[0056] Multi-modal model checks whether the text matches the image (e.g., whether the construction area in the photo matches the description in the manual);
[0057] Dynamic allocation module: generates allocation strategies based on the matching degree of sub-dataset characteristics and the performance of each heterogeneous large model, allocates annotation tasks of multiple large models, and the matching degree is determined by the annotation accuracy and processing speed of the large model on the validation set;
[0058] Annotation accuracy module: detects the F1 score of the large model on the validation set, and the accuracy of the large model during annotation is F1. The higher the F1 value, the more accurate the model, and the maximum F1 value is 100.
[0059] Processing speed module: detects the amount of data processed by the large model per second (e.g., images per second);
[0060] Conflict arbitration module: performs confidence weighting or semantic fusion processing on conflict results generated by different models when annotating the same data. When multiple large models produce different annotations for a set of data, the conflict arbitration module is used for superposition processing. The average accuracy is calculated by dividing the highest accuracy by the sum of the accuracies of multiple large models, such as:
[0061] Model A annotates a picture as "dog" with a confidence of 90%;
[0062] Model B annotates as "wolf" with a confidence of 55%;
[0063] High confidence results have higher weights (e.g., "dog" result weight = 90 / (90+55) = 62%);
[0064] Semantic fusion step: check if the conflict results are essentially the same (e.g., "dog" and "dog"); merge through a thesaurus or knowledge graph (unified as "dog"); if it cannot be merged, submit for manual review;
[0065] Efficient annotation model, used to establish a new large model, which integrates the historical annotation records of multiple large models, and reduces the accuracy to improve the completion speed of different types of annotation tasks;
[0066] Let the new large model learn the annotation output results of multiple historical large models, and then output the annotation task, such as historical large model: 3 large models need to be called (2 seconds per picture);
[0067] After fusion, the new large model: 1 large model completes the same task (0.5 seconds per picture, accuracy decrease <3%);
[0068] Version management module, records operation logs of the annotation process and retains historical records, and preserves historical versions;
[0069] Record the time point of each annotation modification, and keep historical log versions for later viewing;
[0070] In this embodiment, the data splitting module further comprises:
[0071] The primary splitting unit performs coarse-grained division according to data sources or formats, and respectively divides data sets according to significant features, such as mobile phones, computers, and other large categories of data.
[0072] The deep splitting unit merges data with similar semantics into the same subset through a clustering algorithm, can identify similar annotation tasks, and thus merge them into the same subset.
[0073] As preferred, the large model in the multi-model management module comprises at least two combinations of:
[0074] Visual large model: The ViT architecture optimized for image segmentation tasks is input into the Transformer after image blocking, replacing the traditional CNN, and is good at capturing global dependencies.
[0075] Text large model: The sparse attention mechanism Transformer adapted to long text sequence labeling only calculates the attention between key words, reducing the computational complexity.
[0076] Multimodal large model: The joint embedding space model trained by cross-modal contrastive learning maps images and texts to the same space, supporting joint image-text annotation.
[0077] ViT model annotates the crack area in the photo (pixel-level segmentation)
[0078] Text model extracts the "danger warning" keyword in the specification (entity recognition)
[0079] Multimodal model checks whether the image and text match (such as whether the construction area in the photo matches the description in the specification).
[0080] In this embodiment, the dynamic allocation module performs the following operations:
[0081] Data monitoring allocation module: Real-time monitoring of the calculation resource occupancy rate of each model, preferentially allocating high-complexity data to idle models, and the system preferentially matches large models that specialize in processing corresponding sub-data sets. If the workload of the large model corresponding to the sub-data set is overloaded, the task is assigned to a new large model.
[0082] Inefficient large model retraining module: dynamically adjust the allocation weight according to the historical annotation record, trigger the retraining mechanism for the continuously inefficient model, and the large model will be retrained when the annotation error rate on similar data reaches five percent:
[0083] Automatic collection of recent annotation errors, incremental training of the model (fine-tuning, not from scratch) after passing the validation set test, and re-online;
[0084] In this embodiment, the conflict arbitration module further comprises:
[0085] Automatic correction module: used to establish a domain knowledge graph verification channel, verify the semantic rationality of the conflict annotation results, and automatically correct unreasonable annotations;
[0086] Expert module: used to set up an expert voting mechanism, when the degree of disagreement of automatic arbitration exceeds the threshold, trigger online collaborative annotation by domain experts, when multiple models have a large disagreement in tumor benignity classification, decide the final annotation by online voting of three experts.
[0087] An automatic data annotation system and method based on a large model, comprising the following steps:
[0088] Step S1: Dynamic data splitting:
[0089] Through the primary splitting unit in the data splitting module, coarse-grained division is performed according to data sources / formats, and then the deep splitting unit in the data splitting module uses clustering algorithms (such as DBSCAN or K-means) to merge semantically similar data into the same subset, forming a multi-dimensional classification label system;
[0090] Step S2: Multi-model collaborative annotation:
[0091] Call the heterogeneous large models (at least including visual large models, text large models and multi-modal large models) in the multi-model management module, automatically match the optimal model combination according to the characteristics of the sub-data set, and perform annotation tasks in parallel;
[0092] Step S3: Intelligent task allocation:
[0093] The dynamic allocation module calculates the performance indicators of each model on the validation set in real time, dynamically adjusts the task queue in combination with the current GPU utilization, and enables a load balancing strategy for high complexity data;
[0094] Step S4: Conflict semantic fusion:
[0095] When the multi-model annotation results of the same data conflict, a confidence weighted voting mechanism (such as Dempster-Shafer theory) or a semantic consistency verification based on knowledge graph is used to output the fused annotation results;
[0096] Step S5: Incremental model optimization:
[0097] The historical annotation records are taken as training data, and a large model is fine-tuned through an efficient annotation model by using a curriculum learning strategy, so that the annotation speed is improved by 20%-30% while the accuracy is maintained above 90%;
[0098] Step S6: Whole-process version tracing:
[0099] The model version, parameter configuration and operation instruction in the annotation process are recorded, the annotation result of any version can be traced back according to the time stamp, and the difference comparison accuracy reaches 0.01%.
[0100] Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can modify the technical solutions recorded in the foregoing embodiments or make equivalent replacements to some technical features, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A large model-based automated data labeling system, characterized by, Comprise: Data splitting module: for classifying and splitting the original complex data set into multiple sub data sets according to the preset rules, for subsequent multi model management module to carry out labeling processing, the splitting rules include at least one of data type, semantic complexity, labeling task type; Multi model management module: containing multiple heterogeneous large models, each large model is trained for different data characteristics or labeling task, the training mode includes model architecture, training data distribution, labeling granularity, each large model can perform labeling task for different sub data set; Dynamic allocation module: according to the feature of sub data set and the performance matching degree of each heterogeneous large model, generate allocation strategy, allocate the labeling task of multiple large models, the matching degree is quantified by the labeling accuracy, processing speed and field adaptability of large model on the validation set; Conflict arbitration module: the conflict results generated by different models for the same data are processed by confidence weighting or semantic fusion; Efficient labeling model, for establishing new large model, the new large model integrates the historical labeling records of multiple large models, and the accuracy is reduced to improve the completion speed of different types of labeling task; Version management module, record the operation log of labeling process, and keep historical record, keep historical version.
2. The large model-based automated data labeling system of claim 1, wherein: The data splitting module further comprises: Primary splitting unit, coarse-grained division is carried out according to data source or format; Deep splitting unit: similar semantic data is merged into the same subset through clustering algorithm.
3. The large model-based automated data labeling system of claim 1, wherein: The large model in the multi model management module contains at least two combinations: Visual large model: optimize the ViT architecture for image segmentation task; Text large model: adapt sparse attention mechanism transformer for long text sequence labeling; Multi modal large model: joint embedding space model trained by cross modal contrast learning.
4. The large model-based automated data labeling system of claim 1, wherein: The dynamic allocation module further comprises: Data monitoring and allocation module: real-time monitoring of the calculation resource occupation rate of each model, preferentially allocating high complexity data to idle model; Low efficiency large model retraining module: dynamically adjust the allocation weight according to the historical labeling record, and trigger the retraining mechanism for the continuously inefficient model.
5. The large model-based automated data labeling system of claim 1, wherein: The conflict arbitration module further comprises: Automatic correction module: for establishing domain knowledge graph verification channel, verifying the semantic rationality of conflict labeling results, and automatically correcting unreasonable annotations; Expert module: for setting expert voting mechanism, when the automatic arbitration divergence exceeds the threshold, trigger the online collaborative labeling of domain experts.
6. A large model-based automated data labeling method, comprising the large model-based automated data labeling system of any one of claims 1-5, characterized in that: The steps comprise: Step S1: dynamic data splitting: Through the primary splitting unit in the data splitting module, coarse-grained division is carried out according to data source / format, and then the deep splitting unit in the data splitting module adopts clustering algorithm to merge similar semantic data into the same subset, forming a multi-dimensional classification tag system; Step S2: multi model collaborative labeling: Call the heterogeneous large model in the multi model management module, automatically match the optimal model combination according to the characteristics of sub data set, and execute the labeling task in parallel; Step S3: intelligent task allocation: The dynamic allocation module calculates the performance indicators of each model on the validation set in real time, dynamically adjusts the task queue in combination with the current GPU utilization, and enables a load balancing strategy for high complexity data; Step S4: Conflict semantic fusion: When the multi-model labeling results of the same data conflict, a confidence weighting-based voting mechanism or a knowledge graph-based semantic consistency verification is adopted to output the fused labeling results. Step S5: Incremental model optimization: The historical labeling records are used as training data, and the large model is fine-tuned through the course learning strategy of the efficient labeling model, which improves the labeling speed by 20%-30% while maintaining an accuracy rate of over 90%. Step S6: Full-process version tracing: The model version, parameter configuration, and operation instructions in the labeling process are recorded, supporting the tracing of the labeling results of any version according to the timestamp, and the difference comparison accuracy reaches 0.01%.