An AI agent-based intelligent data labeling system and method thereof

Abstract

The application discloses an intelligent data labeling system and method based on an AI Agent, and belongs to the technical field of artificial intelligence data preprocessing. The method comprises the following steps: S1, audio, text and image features of a multi-modal data stream are extracted through a perception module, and a strategy network based on reinforcement learning is used for dynamic distribution to a corresponding labeling task queue; S2, multiple heterogeneous AI Agents are called to perform labeling in parallel, and the confidence of the initial labeling result and the Fleiss' Kappa consistency index between the multiple Agents are calculated; S3, samples lower than a dynamic threshold or with high uncertainty are pushed to an artificial terminal for review, and the artificial terminal generates candidate labels in real-time speech flow by using a pre-input model to provide confirmation and correction; and S4, the labeling data is used for incremental training of a target large model, and the accuracy, recall rate and F1 value of a verification set are used as the criteria. The application realizes adaptive scheduling of the labeling process, real-time human-computer collaboration and continuous evolution of quality, and is suitable for intelligent labeling of multi-modal large model training data such as voice, text and image.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an intelligent data labeling system and method based on an AI agent. Background Technology

[0002] With the rapid development of deep learning and Large Language Model (LLM) technologies, high-quality labeled data has become a key bottleneck restricting the improvement of model performance. Traditional data labeling mainly relies on manual processing of each data item, which has problems such as low efficiency, high cost, and poor consistency, making it difficult to meet the needs of large-scale model training for massive, high-quality, and multimodal data.

[0003] Existing automated annotation tools mostly adopt a fixed-process pipeline architecture, and their annotation models and processing strategies are difficult to dynamically adjust according to the actual data distribution after deployment. When the characteristics of the input data (such as speech signal-to-noise ratio, accent complexity, and image clarity) change significantly, the fixed process often leads to unstable annotation quality, producing a large number of low-confidence results, which in turn increases the workload of manual review in the later stage.

[0004] In the field of speech annotation, especially for spoken pre-input tasks in real-time interactive scenarios, existing technologies lack dedicated processing solutions. Spoken pre-input requires systems to predict complete semantics and provide candidate annotations in real time when users only utter partial words or incomplete sentences, thereby significantly improving annotation efficiency. However, existing platforms typically only support complete post-processing of speech transcription and cannot achieve real-time prediction and interactive annotation during speech stream playback.

[0005] Furthermore, existing annotation platforms generally lack a closed-loop feedback mechanism between annotation quality and downstream model training performance. Adjustments to annotation strategies often rely on manual experience or offline statistics, failing to automatically and continuously optimize the decision parameters and task scheduling strategies of the annotation agent based on the actual performance of large models on validation sets. This makes it difficult for the annotation system to evolve with model iterations.

[0006] Therefore, there is an urgent need for a new type of intelligent data annotation system that can adaptively schedule annotation resources based on data characteristics, support real-time interactive pre-input of spoken language, and has closed-loop self-optimization capabilities. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to overcome the above-mentioned technical difficulties and provide an intelligent data annotation system and method based on AI Agent, which is a novel intelligent data annotation system that can adaptively schedule annotation resources according to data characteristics, support real-time interaction with spoken pre-input, and has closed-loop self-optimization capabilities.

[0008] To address the aforementioned technical problems, the present invention provides a technical solution: an intelligent data annotation method based on an AI Agent, characterized in that the method includes the following steps: S1: Data Feature Awareness and Task Distribution Receive a multimodal data stream to be labeled; extract data features of the multimodal data stream through a perception module, the data features including audio signal-to-noise ratio, speech rate features, text length features, and image sharpness features; based on the data features, distribute the multimodal data stream to the corresponding labeling task queue using a preset policy network; S2: Multi-Agent Collaborative Labeling and Confidence Calculation At least one AI Agent is invoked to perform a labeling task. The AI ​​Agent includes a perception module, a decision-making module, an execution module, and a feedback module. The execution module invokes an adapted labeling model to automatically label the multimodal data stream and generate initial labeling results. The feedback module calculates the confidence score of the initial labeling results. S3: Dynamic Quality Verification and Human-Machine Collaboration Determine whether the confidence score is lower than a preset dynamic threshold; if it is lower than the dynamic threshold, push the initial annotation result and the corresponding multimodal data stream to the manual annotation terminal for manual review and correction to generate high-quality annotation data; if it is not lower than the dynamic threshold, directly use the initial annotation result as valid annotation data. S4: Closed-loop feedback and strategy update The high-quality labeled data and effective labeled data are stored in the labeled dataset; the target large model is incrementally trained using the labeled dataset; the performance evaluation index of the target large model on the validation set is obtained; and the decision parameters of the policy network and the AI ​​Agent are updated in reverse using a reinforcement learning algorithm based on the performance evaluation index.

[0009] As an improvement, step S1, which involves distributing the multimodal data stream to the corresponding annotation task queue based on the data features, specifically includes: Construct a task scheduling model based on reinforcement learning; The data features are input into the task scheduling model as a state space; The task scheduling model outputs an action space, which includes selecting a specific speech recognition model, natural language processing model, or computer vision model as the current annotation model. The speech recognition model includes the Whisper model or the Paraformer model, and the natural language processing model includes a lightweight large language model.

[0010] As an improvement, the multi-Agent collaborative annotation in step S2 specifically includes: For the same multimodal data stream, multiple heterogeneous AI agents are launched to execute annotation tasks in parallel, generating multiple annotation results; Calculate the consistency index among the multiple annotation results, where the consistency index uses the Fleiss' Kappa coefficient; If the consistency index is lower than the preset consistency threshold, it is determined that there is a labeling conflict, and the deep verification process is triggered.

[0011] As an improvement, a sample selection strategy based on active learning is also included: Calculate the uncertainty score of the sample to be labeled, the uncertainty score being calculated based on information entropy or prediction variance; Samples with uncertainty scores higher than a preset screening threshold are prioritized for manual annotation. When receiving a real-time audio stream, the manual annotation terminal uses a pre-input model to generate semantically complete candidate annotations in real time during the audio playback process, and displays them on the interactive interface for annotation personnel to confirm or correct.

[0012] As an improvement, step S4, which involves reversing the policy network based on the performance evaluation metric, specifically includes: Monitor the accuracy, recall, and F1 score of the target large model on the target task; If the improvement in accuracy or F1 score is less than the preset convergence threshold, the active learning screening threshold or the AI ​​Agent's decision-making strategy is adjusted to increase the labeling weight of high-difficulty samples.

[0013] It also includes an AI Agent-based intelligent data annotation system, the system comprising: Data access module: Used to acquire multimodal data to be processed, including voice data, text data and image data.

[0014] Intelligent scheduling module: Used to build the AI ​​Agent environment, dynamically configure and distribute annotation tasks based on data characteristics.

[0015] Collaborative annotation module: Contains multiple AI agents with independent decision-making capabilities, used to execute parallel automatic annotation tasks.

[0016] Human-computer interaction module: Used to provide human annotators with a visual interactive interface and pre-input auxiliary annotation function when the confidence of automatic annotation is insufficient or high uncertainty is detected.

[0017] Closed-loop optimization module: Used to collect annotation results and model training feedback, and dynamically update the policy parameters of the AI ​​Agent and the weights of the labeled model.

[0018] As an improvement, the closed-loop optimization module further includes a semi-supervised learning unit, which is configured as follows: Train a teacher model using labeled data; The teacher model is used to generate pseudo-labels for unlabeled data; Consistency regularization is used to constrain the student model to learn the pseudo-labels, thereby expanding the training dataset and improving the model's robustness.

[0019] The advantages of this invention compared to existing technologies are as follows: Through adaptive scheduling of AI agents and parallel processing of multiple agents, the system can automatically match the optimal annotation resources according to data characteristics, avoiding model overcapacity or undercapacity. Candidate complete sentences are generated during speech playback, allowing annotators to confirm or make minor corrections without listening to the entire audio. Through a closed-loop feedback mechanism, the system uses the performance of downstream large models as an evaluation signal for annotation quality, automatically adjusting agent strategies and active learning thresholds, enabling the annotation system to continuously evolve with model iterations, significantly reducing long-term annotation costs. The system architecture supports multimodal data access such as speech, text, images, and video. The agents and annotation models adopt a pluggable design, which can be flexibly expanded to new modalities or tasks according to business needs, providing a unified, high-quality data annotation infrastructure for large model training. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall architecture of the system of the present invention.

[0021] Figure 2 This is a flowchart of an intelligent data annotation method based on AI Agent according to the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0023] This embodiment provides an intelligent data annotation system based on an AI agent, which includes the following core modules: Data Access Module: Responsible for acquiring multimodal data to be processed. It supports both batch upload and real-time streaming modes. For voice data, it supports formats such as WAV, MP3, and FLAC; for text data, it supports formats such as TXT, JSON, and CSV; and for image data, it supports formats such as JPG, PNG, and WebP. The data access module performs format standardization, deduplication, basic cleaning, and metadata extraction on the raw data.

[0024] Intelligent Scheduling Module: As the "central brain" of the system, it is responsible for building the AI ​​Agent runtime environment and dynamically configuring and distributing labeling tasks based on data characteristics. This module has a built-in policy network (such as a Transformer-based policy model) and maintains the Agent registry and model capability graph.

[0025] The collaborative annotation module comprises multiple heterogeneous AI agents with independent decision-making capabilities (e.g., Agent-A for voice, Agent-B for text, and Agent-C for image). Each agent internally includes a perception submodule, a decision-making submodule, an execution submodule, and a feedback submodule. Agents communicate with each other via a message bus, supporting task negotiation, result broadcasting, and conflict arbitration.

[0026] Human-Computer Interaction Module 104: Provides a visual web interface and API. When the confidence level of automatic annotation is insufficient or high-uncertainty samples are detected, it pushes annotation tasks to human annotators. The interface supports functions such as audio waveform visualization, text highlighting, and image canvas annotation. In particular, in spoken language pre-input scenarios, the interface displays a list of candidate annotations in real time and supports shortcut keys for confirmation / correction.

[0027] The closed-loop optimization module is responsible for collecting annotation results, manual correction records, and model training feedback. It includes an active learning unit, a semi-supervised learning unit, and a policy update unit. The active learning unit maintains an uncertainty sampling queue; the semi-supervised learning unit runs a teacher-student model architecture to generate pseudo-labels; and the policy update unit executes a reinforcement learning algorithm to update the policy network parameters.

[0028] Example 2: Voice Data Annotation Process Using an audio clip of an English conversation with an accent as an example, the annotation process of this invention will be explained in detail: Step S201: Data feature perception and task distribution.

[0029] The user uploads a 5-minute English conversation audio clip with a sampling rate of 16kHz. The data access module performs pre-emphasis, frame segmentation, and noise reduction on the audio. The intelligent scheduling module's perception module extracts the following features: signal-to-noise ratio (SNR) = 12dB (relatively low), speech rate of 145 words per minute (relatively fast), and detection of non-American accent features.

[0030] The policy network takes the above features as state input and outputs the following actions: selects the Whisper-medium model, which has been adaptively fine-tuned by accent, as the main recognizer, selects the Paraformer-large model as the alternative recognizer, and starts the text agent to prepare for subsequent entity recognition tasks.

[0031] Step S202: Multi-Agent Collaborative Labeling.

[0032] The system starts voice agent-A (the main Whisper model) and voice agent-B (the Paraformer model) to perform the transcription task in parallel. Agent-A outputs the transcribed text T1, and Agent-B outputs the transcribed text T2.

[0033] The feedback module calculates the confidence level of T1 as 0.82 and the confidence level of T2 as 0.75. The system calculates the consistency index between T1 and T2 using the Fleiss' Kappa coefficient (here considered a special case of two annotators, i.e., Cohen's Kappa), and calculates κ=0.88, which is higher than the consistency threshold of 0.6. Therefore, it is determined that there is no conflict, and the higher confidence result T1 is taken as the initial annotation.

[0034] Step S203: Dynamic quality verification and human-machine collaboration.

[0035] Since the confidence level of T1 (0.82) is higher than the dynamic threshold of 0.75, the system directly stores T1 as valid labeled data into the temporary dataset without manual verification.

[0036] Meanwhile, the active learning module calculates the uncertainty score of the audio segment. Assuming the segment contains a large number of domain-specific terms (such as professional vocabulary in medical dialogues), the information entropy calculation result is high, and the uncertainty score is 0.68, exceeding the screening threshold of 0.5. Therefore, the system still marks this segment as a "recommended for manual sampling" sample and pushes it to human terminals for quality inspection at a certain proportion (e.g., 5%).

[0037] Step S204: Closed-loop feedback and strategy update.

[0038] All labeled data in this batch (including valid automatically labeled data and data corrected by manual sampling) is stored in the labeled dataset. This dataset is then used to fine-tune the downstream dialogue understanding model using LoRA.

[0039] After fine-tuning, the accuracy of the large model on the validation set improved from 82.3% to 84.1%, and the F1 score improved from 80.5% to 82.8%, both exceeding the convergence threshold (e.g., 1%). The system determined that the current annotation strategy was effective, maintained the existing strategy network parameters, and recorded the mapping relationship between the features of this batch of data and the model selection, serving as an experience replay sample for the strategy network.

[0040] Example 3: Spoken Language Pre-input Scenario Taking real-time annotation of oral test audio in the field of education as an example: Step S301: Real-time voice stream reception.

[0041] The annotator listens to a real-time spoken language test audio through the human-computer interaction module. During the audio stream playback, the system performs real-time speech recognition in 200ms windows and maintains some of the decoding results.

[0042] Step S302: Pre-input candidate generation.

[0043] When the system detects that the current sentence playback progress has reached approximately 2 / 3, or detects a pause of more than 300ms in the speaker's speech, the spoken language pre-input module triggers the prediction mechanism. This module combines the currently recognized text fragment (e.g., "I think the main reason for climate change is…") with an n-gram model and a lightweight LLM (e.g., Qwen-1.8B-Instruct) to generate three semantically complete candidate annotations: Candidate 1: "I believe the primary cause of climate change is greenhouse gas emissions from human activities." Candidate 2: "I believe the main cause of climate change is industrialization and the massive use of fossil fuels." Candidate 3: "I believe the main causes of climate change are deforestation and land-use change." Step S303: Human-computer interaction confirmation.

[0044] Three candidate annotations are displayed in real-time in the candidate list area of ​​the interactive interface. The annotator quickly determines that candidate 1 best matches the speaker's actual semantics and clicks "confirm." If all candidates are inaccurate, the annotator can select the "manual input" mode for complete correction.

[0045] Step S304: Incremental model update.

[0046] The system records the annotations' selections and corrections. Every Friday, the system uses all the pre-input correction data accumulated that week to perform incremental training on the lightweight LLM, enabling it to learn language patterns specific to a particular domain (such as educational oral tests) and improve the accuracy of pre-input for the following week.

[0047] Example 4: Multi-Agent Conflict Detection and Deep Verification Take a multimodal meeting record that simultaneously contains voice, text, and images as an example: Step S401: Multimodal parallel annotation.

[0048] The system receives a meeting record data package, which includes: meeting audio recording (voice), scanned copy of meeting minutes (image), and a list of attendees (text). The intelligent scheduling module distributes tasks: the voice agent performs transcription, the OCR agent performs image text recognition, and the text agent performs name alignment and entity recognition.

[0049] Step S402: Result consistency verification.

[0050] The speech agent identified the speaker as "John Smith" in the transcription results, the OCR agent identified "Jon Smith" from the image (the handwriting caused confusion between "o" and "oh"), and the text agent confirmed the name as "John Smith" from the list of attendees.

[0051] The system detected inconsistencies in the speaker name labeling among the three agents. The Fleiss' Kappa coefficient was calculated to be κ=0.35, which is below the consistency threshold of 0.6, triggering a deep verification process.

[0052] Step S403: Deep verification and manual arbitration.

[0053] The system initiates Arbitration Agent-D, which calls a joint verification model that integrates phoneme features and glyph features to comprehensively analyze the acoustic features of the speech segment and the local features of the image, and outputs the arbitration result: "John Smith" is correctly labeled, and "Jon Smith" is a recognition error in the OCR result.

[0054] The arbitration result had a confidence level of 0.91, which was higher than the threshold. The system automatically corrected the OCR Agent's annotation results and fed back the erroneous sample and correction record to the OCR Agent's execution module for online model learning. Simultaneously, the conflict case was recorded in the conflict case database for subsequent training of a dedicated conflict detection model.

[0055] Example 5: Semi-supervised learning and data augmentation Suppose that in a certain image annotation project, there are only 5,000 labeled images and 50,000 unlabeled images.

[0056] Step S501: Teacher model training.

[0057] The semi-supervised learning unit of the closed-loop optimization module trains a teacher model (such as ResNet-50) using 5000 labeled images, which achieves 85% accuracy on the validation set.

[0058] Step S502: Pseudo-tag generation.

[0059] The teacher model generates pseudo-labels for 50,000 unlabeled images. Pseudo-label samples with a prediction confidence score higher than 0.95 are directly included in the training set; samples with a confidence score between 0.8 and 0.95 are included in the training set at a certain proportion (e.g., 30%); samples with a confidence score lower than 0.8 are not used for the time being.

[0060] Step S503: Consistency regularization training.

[0061] Student models (such as MobileNet-V3, used for edge deployment) learn the aforementioned pseudo-labeled data. During training, random data augmentation (cropping, rotation, color jitter) is applied to the input images, and consistency regularization constraints require the student to maintain consistency between the output of the original image and the augmented image (i.e., minimizing L2 distance or KL divergence).

[0062] Step S504: Model iteration.

[0063] The trained student model achieved an accuracy of 83% on the validation set, approaching the level of the teacher model. This student model can be deployed as a lightweight annotation model to an agent in the collaborative annotation module for rapid pre-annotation, thereby significantly reducing the reliance on high computing resources.

[0064] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. An intelligent data annotation method based on AI Agent, characterized in that, The method includes the following steps: S1: Data Feature Awareness and Task Distribution Receive a multimodal data stream to be labeled; extract data features of the multimodal data stream through a perception module, the data features including audio signal-to-noise ratio, speech rate features, text length features, and image sharpness features; based on the data features, distribute the multimodal data stream to the corresponding labeling task queue using a preset policy network; S2: Multi-Agent Collaborative Labeling and Confidence Calculation At least one AI Agent is invoked to perform a labeling task. The AI ​​Agent includes a perception module, a decision-making module, an execution module, and a feedback module. The execution module invokes an adapted labeling model to automatically label the multimodal data stream and generate initial labeling results. The feedback module calculates the confidence score of the initial labeling results. S3: Dynamic Quality Verification and Human-Machine Collaboration Determine whether the confidence score is lower than a preset dynamic threshold; if it is lower than the dynamic threshold, push the initial annotation result and the corresponding multimodal data stream to the manual annotation terminal for manual review and correction to generate high-quality annotation data; if it is not lower than the dynamic threshold, directly use the initial annotation result as valid annotation data. S4: Closed-loop feedback and strategy update The high-quality labeled data and effective labeled data are stored in the labeled dataset; the target large model is incrementally trained using the labeled dataset; the performance evaluation index of the target large model on the validation set is obtained; and the decision parameters of the policy network and the AI ​​Agent are updated in reverse using a reinforcement learning algorithm based on the performance evaluation index.

2. The method according to claim 1, characterized in that, Step S1, which involves distributing the multimodal data stream to the corresponding annotation task queue based on the data features, specifically includes: Construct a task scheduling model based on reinforcement learning; The data features are input into the task scheduling model as a state space; The task scheduling model outputs an action space, which includes selecting a specific speech recognition model, natural language processing model, or computer vision model as the current annotation model. The speech recognition model includes the Whisper model or the Paraformer model, and the natural language processing model includes a lightweight large language model.

3. The method according to claim 1, characterized in that, The multi-agent collaborative labeling in step S2 specifically includes: For the same multimodal data stream, multiple heterogeneous AI agents are launched to execute annotation tasks in parallel, generating multiple annotation results; Calculate the consistency index among the multiple annotation results, where the consistency index uses the Fleiss' Kappa coefficient; If the consistency index is lower than the preset consistency threshold, it is determined that there is a labeling conflict, and the deep verification process is triggered.

4. The method according to claim 1, characterized in that, The dynamic quality verification in step S3 also includes a sample selection strategy based on active learning: Calculate the uncertainty score of the sample to be labeled, the uncertainty score being calculated based on information entropy or prediction variance; Samples with uncertainty scores higher than a preset screening threshold are prioritized for manual annotation. When receiving a real-time audio stream, the manual annotation terminal uses a pre-input model to generate semantically complete candidate annotations in real time during the audio playback process, and displays them on the interactive interface for annotation personnel to confirm or correct.

5. The method according to claim 1, characterized in that, Step S4, which involves reversing the policy network based on the performance evaluation metric, specifically includes: Monitor the accuracy, recall, and F1 score of the target large model on the target task; If the improvement in accuracy or F1 score is less than the preset convergence threshold, the active learning screening threshold or the AI ​​Agent's decision-making strategy is adjusted to increase the labeling weight of high-difficulty samples.

6. An intelligent data annotation system based on AI Agent, characterized in that, The system for implementing the method as described in any one of claims 1 to 5 comprises: Data access module: Used to acquire multimodal data to be processed, including voice data, text data and image data. Intelligent scheduling module: Used to build the AI ​​Agent environment, dynamically configure and distribute annotation tasks based on data characteristics. Collaborative annotation module: Contains multiple AI agents with independent decision-making capabilities, used to execute parallel automatic annotation tasks. Human-computer interaction module: Used to provide human annotators with a visual interactive interface and pre-input auxiliary annotation function when the confidence of automatic annotation is insufficient or high uncertainty is detected. Closed-loop optimization module: Used to collect annotation results and model training feedback, and dynamically update the policy parameters of the AI ​​Agent and the weights of the labeled model.

7. The system according to claim 1, characterized in that, The closed-loop optimization module further includes a semi-supervised learning unit, which is configured as follows: Train a teacher model using labeled data; The teacher model is used to generate pseudo-labels for unlabeled data; Consistency regularization is used to constrain the student model to learn the pseudo-labels, thereby expanding the training dataset and improving the model's robustness.

Images