Interactive data labeling and model training method based on large model agent

Abstract

The application relates to the technical field of artificial intelligence, and discloses an interactive data labeling and model training method based on a large model intelligent agent. The method constructs a multi-level fuzzy comprehensive evaluation requirement ambiguity quantification system, identifies multi-dimensional ambiguity through entropy weight dynamic weighting, and realizes cognitive alignment by clarifying according to contribution degree; combines small sample interactive labeling, automatic training and active guided optimization, matches full-link context memory, constructs a deep human-computer collaborative closed loop, and realizes local deployment in the whole process to guarantee data security. The application improves ambiguity recognition depth and interaction efficiency, reduces model production threshold and rework cost, and shortens the model iteration cycle.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to an interactive data annotation and model training method based on large model intelligent agents. Background Technology

[0002] With the widespread application of artificial intelligence technology in industries, security and other fields, the demand for customized AI models for specific scenarios continues to grow. Currently, AI model production mainly adopts two technical routes.

[0003] One approach is the traditional manual annotation and training scheme. This involves collecting raw video and image data using cameras and sensors, followed by manual preprocessing operations such as video segmentation, frame extraction, deduplication, and cropping. Annotators then manually draw bounding boxes, label action sequences, and classify data categories. Algorithm engineers then write training code and manually adjust hyperparameters such as learning rate, batch size, and iteration count to complete model training. Finally, the model's accuracy is manually tested, and error samples are analyzed. Multiple rounds of re-annotation and parameter tuning are conducted until the model reaches the required accuracy. This scheme relies entirely on manual operation, resulting in extremely low annotation efficiency. Annotating a single industrial scenario dataset typically takes weeks or even months and is prone to inconsistencies, omissions, and errors. Furthermore, model training and parameter tuning require a deep understanding of deep learning theory and engineering experience, which frontline business personnel cannot participate in. The model's lifecycle from data collection to deployment is lengthy, and each project requires a dedicated annotation and algorithm team, leading to high labor costs. The large amount of repetitive annotation and parameter tuning work stifles creativity, and model quality heavily depends on the individual experience of algorithm engineers, resulting in inconsistent quality across different projects.

[0004] Another approach is general-purpose training software solutions. Some general-purpose AI training software on the market provides basic annotation and training functions, but only offers basic rectangular and polygon annotation tools without intelligent assistance functions. All annotation work still needs to be done manually. At the same time, the training process has low transparency, and users cannot intuitively understand the training process. Parameter tuning relies entirely on experience, and all operations must be completed through graphical interfaces or code. It is impossible to express requirements through natural language. Model optimization requires repeating the entire process of annotation-training-testing, resulting in low iteration efficiency and a high barrier to entry. It must be operated by algorithm engineers with deep learning knowledge, and front-line business personnel cannot participate. Even though some solutions incorporate natural language interaction capabilities, they are all passive, one-way command interactions. They can only identify the clarity of user needs through simple entity extraction. Their ambiguity determination is based solely on the confidence weighting of single-dimensional entities, failing to identify rule conflicts between dimensions, ambiguous ranges of scene boundaries, and implicit cognitive biases between users and the system. This results in a large number of implicit ambiguities not being discovered in advance, often only revealing requirement deviations after annotation or model training, requiring rework and correction, which reduces overall efficiency. At the same time, multi-turn interactions lack contextual memory capabilities, requiring users to repeatedly describe business rules and requirements at different stages such as annotation and optimization, resulting in low interaction efficiency. Furthermore, they cannot proactively guide users to complete complex operations such as error analysis and optimization adjustments. Users still need to locate model problems themselves, failing to achieve true deep human-machine collaborative interaction, further limiting the user experience for ordinary business personnel and the efficiency of model production. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides an interactive data annotation and model training method based on a large-scale intelligent agent, employing the following technical solution: An interactive data annotation and model training method based on large model agents includes the following steps: S1. Receive the raw multimodal data uploaded by the user and obtain the initial annotation and training requirements input by the user through natural language. S2. The initial requirements are analyzed by the annotation agent. Based on the pre-constructed four-level ambiguity evaluation index system, the ambiguity membership degree of each dimension is calculated by fuzzy membership function. The overall requirement ambiguity degree is calculated by weighted fusion. When the ambiguity degree exceeds the preset threshold, a hierarchical clarification interaction is initiated according to the hierarchical order of "basic dimension - cross-dimensional association - scene boundary - implicit hypothesis" based on the level of the ambiguity type and its logical dependency. After confirmation, structured annotation rules are generated. Data annotation is completed based on the few-shot learning mechanism and multi-round interactive confirmation, and an annotated dataset is generated. S3. By training the intelligent agent, the training requirements are analyzed, the basic model and hyperparameter configuration are automatically matched, local computing resources are scheduled to execute model training, and the training status and abnormal warnings are proactively fed back to the user during the training process. S4. Evaluate the performance of the trained model by evaluating the agent, classify and count error samples, and generate evaluation results; S5. The optimization agent proactively pushes error classification and optimization direction guidance to the user based on the evaluation results. After receiving optimization feedback from the user through natural language input, it performs targeted data augmentation and incremental fine-tuning to complete the model iteration. S6. Export the model file that meets the performance requirements to the target device.

[0006] Optionally, in step S2, the calculation of the overall demand ambiguity specifically includes: Construct a four-level ambiguity evaluation index system: the target level is the comprehensive demand ambiguity degree D; Criterion layer is the basis for dimensional ambiguity Cross-dimensional association ambiguity Ambiguity of scene boundaries Implicit assumption bias Each criterion layer has a corresponding indicator layer. The ambiguous membership degree of each individual indicator is calculated using a pre-set fuzzy membership function. Specifically, a descending half-trapezoidal membership function is used for confidence level indicators, while an ascending half-trapezoidal membership function is used for positive indicators such as rule conflict degree and missing degree. Criterion layer weights pre-set by expert experience and indicator layer weights By performing weighted fusion at each level, the overall ambiguity of demand is obtained; Simultaneously, the ambiguity contribution of each indicator is calculated, and the clarification order is determined according to the logical dependency relationship based on the level to which the ambiguity type belongs. Basic dimension ambiguity is dealt with first, followed by cross-dimensional correlation ambiguity, scenario boundary ambiguity, and implicit assumption bias.

[0007] Optionally, after the labeling agent guides the user to complete the initial sample labeling that is adapted to the task complexity, it completes batch pre-labeling based on the small sample inference capability of the edge-side lightweight multimodal base model. During the pre-labeling process, it identifies boundary samples with confidence scores lower than the system's preset confidence threshold, actively pushes them to the user for confirmation, and receives the user's natural language error correction instructions to correct the results and update the labeling rules and sample example library.

[0008] Optionally, step S3 specifically includes: The training agent analyzes the task type, accuracy target, and data attributes in the natural language training requirements, matches the appropriate base model from the preset model library, automatically generates the optimal hyperparameter combination based on the Bayesian optimization algorithm, schedules the local distributed computing power cluster to execute model training, collects training indicators in real time, provides feedback to the user on training progress and convergence status through natural language, and proactively issues warnings and provides adjustment suggestions when training anomalies are detected.

[0009] Optionally, step S5 specifically includes: The evaluation agent performs feature clustering and error type classification on erroneous samples, and the tuning agent pushes the error classification and optimization direction to the user in natural language to guide the user to confirm the optimization direction. After receiving user feedback, the model identification deviation is located, corresponding data augmentation strategies are generated and training samples are supplemented. The model is optimized by using layer freeze incremental fine-tuning to quickly complete the model iteration.

[0010] Optionally, a full-link interactive context memory pool is constructed to store users' historical annotation standards, judgment boundaries, optimization preferences and task objectives. Context information is automatically reused in multi-round interactions and cross-stage tasks to calculate the implicit hypothesis bias and avoid users repeatedly describing their requirements. During the system initialization phase, when the context memory pool is empty, the implicit hypothesis bias is processed using the default value.

[0011] Optionally, the entire process of data annotation, model training, evaluation and optimization is executed in the local computing power device cluster. Data and model files do not flow out of the local environment. At the same time, fine-grained user permission control and full version traceability management of datasets are configured. Dataset version management adopts incremental storage method, only recording the difference data of each modification to control storage costs.

[0012] Optionally, the multimodal data includes image, video, and audio data, and the supported annotation types include object detection annotation, action recognition annotation, semantic segmentation annotation, and anomaly detection annotation.

[0013] Optionally, the Bayesian optimization algorithm uses a Gaussian process as a surrogate model and iteratively selects the optimal hyperparameter combination by improving the expected value acquisition function to adapt to the parameter requirements of the current training task. The number of hyperparameter optimization iterations is dynamically configured according to the amount of task data and computing resources.

[0014] Optionally, the labeling agent, training agent, evaluation agent, and tuning agent are all encapsulated based on a lightweight multimodal base model on the edge. Each agent interacts with data and collaborates on tasks through a local application programming interface, sharing a unified interaction context memory pool.

[0015] Optionally, the initial number of labeled samples for few-shot learning in step S2 is dynamically determined by the labeling agent based on the task type, the number of categories, and the few-shot reasoning ability of the base model. Specifically, for binary object detection tasks, the initial number of samples is 5-8; for multi-class (number of categories ≥ 3) tasks, the initial number of samples is 3-5 per category; and for semantic segmentation tasks, the initial number of samples is 8-12.

[0016] Optionally, the preset threshold in step S2 is configured to 0.3 during system initialization and can be adaptively adjusted based on user historical interaction feedback; the confidence threshold is configured to 0.6 during system initialization and can be configured separately for different task types.

[0017] Optionally, the number of clarification questions in a single round of the hierarchical clarification interaction shall not exceed 3, and the total number of interaction rounds shall be capped at 8. When the ambiguity is not eliminated after 6 rounds of clarification, the labeling agent automatically switches strategies and provides candidate labeling rule templates for the user to choose from. When the maximum of 8 rounds is reached, the currently confirmed rule shall be used as the standard, and a list of ambiguities to be reviewed shall be generated for the user to confirm before entering the labeling process.

[0018] Optionally, the specific judgment criteria for the training agent to detect training anomalies are as follows: using a sliding window statistical method, with a window size of 10 training steps, the upward trend strength of the training loss value within the window is calculated. When the upward trend strength exceeds a preset threshold and the validation set accuracy decreases synchronously beyond the tolerance threshold, an anomaly warning is triggered. The tolerance threshold is dynamically calculated by the training agent based on the current training round and the task accuracy target.

[0019] Optionally, the reuse of historical context in the context memory pool must meet the similarity judgment condition: before reusing historical context, the labeled agent first calculates the semantic similarity between the current task requirement and the historical task. When the similarity exceeds the dynamic threshold, the corresponding historical context is reused; otherwise, the current task requirement takes precedence.

[0020] Optionally, for scenes where the labeled object in the video data undergoes significant deformation, occlusion, or rapid movement, the labeling agent uses an optical flow tracing algorithm to assist in generating labeling candidates for non-key frames after labeling keyframes. The mapping between optical flow tracing and the label box is determined by calculating the mean optical flow vector of feature points within the label box to determine the bounding box displacement, and by estimating the bounding box scale scaling through changes in the feature point distribution scale. Optical flow tracing is reset every N frames based on keyframes, where N is dynamically determined by the labeling agent according to the video frame rate and the intensity of motion. Frames with optical flow tracing confidence below a threshold are pushed to the user for confirmation.

[0021] In summary, this application includes at least one of the following beneficial technical effects: This invention encapsulates the capabilities of a large model into multiple types of functional intelligent agents, constructing a proactive, collaborative, multi-round interaction system. This breaks the limitations of traditional passive, command-based interaction. The ambiguity calculation system for multi-level fuzzy comprehensive evaluation breaks through the simple judgment mode of existing single-dimensional confidence weighting. It combines a dual-mode weighting mechanism of pre-set expert weights and dynamic entropy weighting. In cold-start scenarios, it leverages expert experience to ensure basic rationality; in scenarios with existing data accumulation, it uses the entropy weighting method for objective and adaptive weighting, enabling more accurate quantification of the degree of ambiguity in requirements and pinpointing core points of ambiguity. This, combined with a hierarchical clarification mechanism... This system achieves precise cognitive alignment, significantly reducing annotation rework and model deviation caused by implicit ambiguity, and improving interaction efficiency and accuracy of requirement restoration. The intelligent agent can proactively push boundary samples, provide feedback on training anomalies, and guide the direction of optimization, eliminating the need for users to locate and sort out problems themselves. This significantly reduces the requirements for users' ability to express their needs and their professional knowledge. Combined with the full-link interactive context memory capability, it enables cross-stage requirement reuse, avoids repetitive descriptions, further improves interaction efficiency, and truly realizes a human-machine collaborative working mode. This allows front-line business personnel to independently complete the entire process from data annotation to model optimization, significantly reducing the technical threshold for AI model production. This invention utilizes an interactive intelligent annotation mechanism to automatically pre-annotate batches of data using only a small number of samples. Combined with natural language error correction and iterative optimization of annotation results, it significantly reduces the workload of manual annotation, improves annotation efficiency, and ensures annotation consistency through unified annotation rules. By automatically completing model selection, hyperparameter configuration, and training execution through an intelligent agent, and with targeted tuning driven by natural language feedback, it constructs an automated closed loop of annotation-training-evaluation-optimization. This significantly shortens the model iteration cycle, enabling rapid response to changes in business scenarios, while reducing the need for specialized algorithm personnel and lowering labor costs. Automated parameter configuration and iterative optimization also avoid model quality fluctuations caused by differences in human experience, improving the stability of model output. The end-to-end localized processing architecture ensures data and model security, avoiding the risk of data leakage, and frees personnel from repetitive, mechanical tasks, allowing them to focus on creative work such as business requirement analysis and model performance evaluation. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the basic process of the interactive data annotation and model training method in this invention. Detailed Implementation

[0023] The embodiments of this application are described in detail below, and examples of the embodiments are shown in the accompanying drawings.

[0024] In the description of this specification, the references to "certain embodiments," "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples" refer to specific features, structures, materials, or characteristics described in connection with the described embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0025] The system upon which this invention is based adopts a three-layer architecture, including a foundation layer, an intelligent agent layer, and a visual interaction layer, wherein: The basic layer includes a data storage module, a distributed training engine, a model management module, and a device access module. The data storage module is used to store raw data, labeled datasets, and model files. The distributed training engine is used to schedule local cluster computing resources to execute parallel training tasks. The model management module is used to manage the basic model library and the model versions generated by training. The device access module is used to connect to the computing power devices and data acquisition devices on the end side. The agent layer is the core of the system, which includes four types of functional agents: labeling agents, training agents, evaluation agents, and tuning agents. Each agent is encapsulated based on a lightweight multimodal base model on the edge and interacts with the base layer module through a local application programming interface. At the same time, they share a unified interaction context memory pool and work together to complete the entire process task. The visualization interaction layer provides a natural language dialogue interface and a visual operation interface to receive user natural language commands and inputs, while displaying visual information such as training progress and evaluation results. It also carries interactive content proactively pushed by the agent, including clarification questions, anomaly warnings, and optimization guidance. Communication between the agent and the visualization interaction layer uses a message queue for asynchronous pushes. After generating push content, the agent writes it to the message queue. The visualization interaction layer consumes the push content from the message queue via a long-lived connection (WebSocket) and displays it to the user in real time, ensuring that the user receives proactive push information from the agent without needing to manually refresh the page.

[0026] In this embodiment, the edge-side lightweight multimodal base model adopts an INT8-quantized Transformer architecture, which includes a visual encoder and a text encoder. The visual encoder adopts a lightweight residual network structure, and the text encoder adopts a lightweight Transformer encoder. The two achieve feature alignment through a cross-modal attention layer. The model has been optimized for edge computing power through structured pruning and knowledge distillation, which can complete low-latency inference and small sample feature matching on edge devices. Full model training is scheduled to be executed by a local distributed computing power cluster, which ensures that the data does not leave the local environment throughout the process, while also taking into account the training computing power requirements.

[0027] The interaction context memory pool adopts a structured storage architecture, divided into three categories: task global context, stage interaction context, and user preference context. The task global context stores information such as the overall task objective and accuracy requirements; the stage interaction context stores information such as annotation rules and optimization parameters for each stage; and the user preference context stores information such as judgment criteria and optimization tendencies in the user's historical operations. All agents can read and call these information, enabling cross-stage information reuse. During system initialization, the context memory pool is empty, and the implicit hypothesis bias is handled with maximum uncertainty, i.e., a value of 1.0. A conservative strategy is used to trigger clarification, i.e., to proactively clarify implicit hypotheses to the user to avoid rework due to inappropriate default assumptions. The context memory pool is gradually updated to the actual calculated values ​​as user interactions accumulate.

[0028] For the reuse of historical context, before reading the historical context, the annotation agent first judges the degree of association between the current task and the historical task based on the semantic vector similarity. The similarity is obtained by calculating the cosine similarity after encoding the task description into a semantic vector by a pre-trained text encoder. The corresponding historical annotation rules and judgment boundaries are reused only when the similarity exceeds a preset threshold (e.g., 0.7) to avoid erroneous knowledge transfer between different tasks.

[0029] like Figure 1 The following is a basic overview of the interactive data annotation and model training method of this invention: S1. Receive the raw multimodal data uploaded by the user and obtain the initial annotation and training requirements input by the user through natural language. S2. The initial requirements are parsed by the labeling agent, and the ambiguity of the comprehensive requirements is calculated based on the multi-level fuzzy comprehensive evaluation model. When the ambiguity exceeds the preset threshold, the hierarchical clarification interaction is initiated actively according to the ambiguity contribution. After confirmation, the structured labeling rules are generated. The data labeling is completed based on the few-shot learning mechanism and multiple rounds of interactive confirmation, and the labeled dataset is generated. S3. By training the intelligent agent, the training requirements are analyzed, the basic model and hyperparameter configuration are automatically matched, local computing resources are scheduled to execute model training, and the training status and abnormal warnings are proactively fed back to the user during the training process. S4. Evaluate the performance of the trained model by evaluating the agent, classify and count error samples, and generate evaluation results; S5. The optimization agent proactively pushes error classification and optimization direction guidance to the user based on the evaluation results. After receiving optimization feedback from the user through natural language input, it performs targeted data augmentation and incremental fine-tuning to complete the model iteration. S6. Export the model file that meets the performance requirements to the target device.

[0030] In this embodiment, the interactive annotation process of the present invention is as follows: After users upload raw image or video data through the visual interaction layer, they input their natural language annotation requirements. The annotation agent first calls the interaction context memory pool to search for whether there are annotation rules and user preferences for similar historical tasks (which must meet the semantic similarity threshold condition). If they exist, they are used as implicit hypothesis benchmarks for subsequent ambiguity calculations.

[0031] Subsequently, the annotation agent performs semantic parsing on the natural language text, completing intent recognition and entity extraction through a pre-trained semantic recognition model, and then performing multi-level fuzzy comprehensive evaluation to calculate ambiguity. The semantic parsing model outputs its own confidence score along with the parsing results. When the parsing confidence score is below 0.6, the annotation agent first confirms the basic accuracy of the parsing results with the user, and then calculates ambiguity based on the confirmed results to avoid inaccurate ambiguity quantification due to semantic parsing errors.

[0032] In this embodiment, the specific steps for calculating the overall demand ambiguity are as follows: The first step is to construct a four-level ambiguous evaluation index system: Target layer: Overall requirement ambiguity D; Criterion Layer: Ambiguity in Basic Dimensions Cross-dimensional association ambiguity Ambiguity of scene boundaries Implicit assumption bias ; Indicator Layer: Each criterion layer corresponds to a sub-indicator, where the basic dimensions are ambiguous. Includes target category confidence Confidence level of the decision boundary Scope of application and confidence level Confidence level of excluded scenarios Cross-dimensional association ambiguity Includes rule conflict degree Used to measure the degree of logical contradiction between rules of different dimensions; ambiguity at scene boundaries. Includes granular ambiguity Quantization threshold missing degree These measures respectively assess the degree of fuzziness in labeling granularity and the degree of lack of quantifiable judgment criteria; implicit hypothesis bias. Includes historical context bias Common sense cognitive bias They respectively measure the degree of deviation between current needs and historical task rules, and the degree of deviation from general industry cognitive standards.

[0033] The numerical sources for the above indicators are as follows: Target category confidence, decision boundary confidence, applicable scope confidence, and excluded scenario confidence are obtained by the semantic parsing model scoring the entity completeness and explicitness of the user input; rule conflict is obtained by verifying the consistency of multi-dimensional rules in the user's requirements through a pre-built logical conflict detection rule base; granular ambiguity and quantification threshold missingness are evaluated by detecting whether the user input contains operable quantification criteria (such as specific numerical thresholds or ranges); historical context bias is obtained by calculating the semantic vector distance between the current requirement and historical tasks in the memory pool; common sense perception bias is obtained by comparing the current requirement with the standard definition in the industry knowledge graph.

[0034] The second step employs a dual-mode weighting mechanism combining pre-set expert weights and dynamic weighting using the entropy weighting method. For scenarios with historical task data or similar reference datasets, the entropy weighting method dynamically calculates the weights of each indicator. First, the characteristics of the current task dataset are statistically analyzed to obtain the sample distribution information of each indicator, constructing an evaluation matrix. Then, the entropy value of each indicator is calculated using information entropy, thus obtaining the objective weight of each indicator. (The last part, "Criterion layer weights," appears to be an unrelated technical term and is omitted from the translation.) Indicator layer weights The formulas for calculating entropy and weight are as follows: ; ; In the formula, Let k be the information entropy of the j-th indicator, reflecting the degree of information disorder of that indicator. The lower the entropy value, the higher the indicator's discriminative power and the greater its weight; k is a constant. m is the number of evaluation samples; This represents the normalized percentage of the i-th sample on the j-th indicator. Let be the weight coefficient of the j-th indicator, and the sum of all weights is 1. Dynamic weighting using the entropy weighting method automatically adjusts the importance of each ambiguous dimension according to the data characteristics of different tasks, avoiding the subjectivity of fixed weights and adapting to the ambiguity sensitivity of different annotation tasks such as object detection and semantic segmentation. For cold start scenarios (i.e., situations lacking historical task data and similar reference datasets), weighted fusion is performed using pre-set weights based on domain expert experience to ensure the system still has reasonable ambiguity quantification capabilities in the initial stage. Pre-set weights can be set based on expert experience; for example, for general object detection tasks, the criterion layer weights can be set to... The weights of the indicator layer are evenly distributed within the corresponding criterion layer; the specific values ​​of the preset weights can be adjusted based on expert experience for different task domains.

[0035] The third step is to calculate the ambiguity degree of membership for each individual indicator.

[0036] For confidence-based indicators (such as target category confidence), a reduced semi-trapezoidal fuzzy membership function is used: ; For positive indicators such as rule conflict degree and missing degree, an ascending semi-trapezoidal fuzzy membership function is used: ; In the formula, _x_ represents the ambiguity degree of the indicator x, with higher values ​​indicating stronger ambiguity; _a_ and _b_ are threshold parameters of the membership function, set based on the statistical distribution of historical annotation tasks for different indicators, and adjusted online according to user feedback during system operation. For example, for the target category confidence indicator, _a_ is 0.7 and _b_ is 0.9; for the rule conflict indicator, _a_ is 0.2 and _b_ is 0.6.

[0037] The fourth step is to obtain the overall requirement ambiguity by weighted fusion layer by layer. First, calculate the ambiguity value for each criterion layer: ; In the formula, The ambiguity value of the i-th criterion layer; Let be the weight of the j-th indicator under the i-th criterion layer; This represents the ambiguity degree of the corresponding indicator. The ambiguities of the four criterion levels are then weighted and fused to obtain the final comprehensive requirement ambiguity degree: ; In the formula, D represents the degree of ambiguity of comprehensive demand, with a value ranging from 0 to 1; is the weight coefficient of the i-th criterion layer.

[0038] The fifth step is to calculate the ambiguity contribution of each indicator, using the following formula: ; In the formula, The ambiguity contribution of the j-th indicator in the i-th criterion layer represents the proportion of the influence of this indicator on the overall ambiguity.

[0039] When the calculated overall ambiguity D exceeds the system's preset threshold (initial value set at 0.3, adjustable adaptively based on user feedback), proactive clarification interaction is triggered. The clarification order follows a logical dependency: prioritizing clarification of basic dimension ambiguities (e.g., unclear target category), then addressing cross-dimensional correlation ambiguities (e.g., rule conflicts), scenario boundary ambiguities (e.g., missing quantization thresholds), and finally handling implicit hypothesis biases. The annotation agent generates natural language clarification questions for the current level of ambiguity indicators and pushes them to the user. After receiving user feedback, it updates the corresponding indicator values, recalculates the overall ambiguity, and repeats the process until the ambiguity falls below the threshold or the number of interaction rounds reaches the preset limit. In this layered clarification interaction, the number of clarification questions per round does not exceed 3 to avoid overwhelming the user with too many questions at once; the total number of interaction rounds is capped at 8. When 6 rounds of clarification have passed and the ambiguity remains unresolved, the annotation agent automatically switches its strategy from question mode to recommendation mode, providing 2-3 candidate annotation rule templates for the user to choose from, thus accelerating the clarification process. When the maximum of 8 rounds is reached, a list of ambiguities to be reviewed (containing unresolved ambiguities and their possible interpretations) is generated based on the currently confirmed rules for the user's final confirmation. After the user's confirmation, the annotation process begins. Through layered and progressive clarification, only the most critical ambiguities are addressed each time, avoiding the burden on the user by raising too many questions at once. At the same time, implicit ambiguities are accurately located, reducing rework in subsequent annotation and training.

[0040] After the rules are confirmed, the annotation agent randomly selects samples from the raw data that are appropriate for the task complexity and pushes them to the user for annotation. The initial number of annotation samples is dynamically determined by the annotation agent based on the task type, the number of categories, and the few-shot inference capability of the base model: for binary object detection tasks, the initial number of samples is 5-8; for multi-class (≥3 categories) tasks, the initial number of samples is 3-5 per category; for semantic segmentation tasks, the initial number of samples is 8-12. The above numerical ranges are based on statistical analysis of few-shot inference experiments of the base model on public datasets. In specific experiments, for binary object detection tasks, 5-8 samples can enable the model to achieve an initial annotation accuracy of over 85%; for multi-class tasks, 3-5 samples per category can enable the model to achieve an initial annotation accuracy of over 80%; and for semantic segmentation tasks, 8-12 samples can enable the model to achieve an initial annotation accuracy of over 82%. After the user completes a small number of sample annotations, the annotation agent performs few-shot learning on the lightweight multimodal base model at the sample input end. Specifically, it constructs few-shot inference prompts through prompt word engineering, inputs the already annotated samples as example context into the base model, and the base model performs feature extraction and matching inference on the remaining unannotated data, outputting the bounding box coordinates and category label for each sample, completing the batch pre-annotation. During the pre-annotation process, the annotation agent calculates the annotation confidence of each sample in real time. When the confidence is lower than the system's preset confidence threshold (initial value set to 0.6, which can be configured for different task types), it is determined to be a boundary sample. This type of sample is actively pushed to the user for confirmation in batches. The user can directly confirm or input correction commands through natural language. The annotation agent parses and corrects the annotation results, and at the same time adds the corrected samples to the example set to update the annotation rules. In subsequent pre-annotation, the corrected standards are automatically adapted to continuously improve the annotation accuracy.

[0041] The parsing of natural language error correction instructions employs the following method: First, the error correction object is located through a referential resolution mechanism—dependency parsing is performed on the user input to identify demonstrative pronouns (such as "this" and "that") and limiting descriptions (such as "the box on the left" and "the second person"). Combined with the context of the labeled samples displayed on the current interface, the instruction is mapped to the specific labeled sample and labeled box. Second, the correction type is identified through error correction intent classification—based on a pre-trained text classification model, user error correction instructions are divided into three categories: position correction, category correction, and judgment criterion correction. Finally, the error correction instruction is mapped to the specific labeled sample for correction through labeled sample association matching. For instructions involving judgment criterion correction (such as "this is not considered a missed detection"), in addition to correcting the current sample, the corresponding judgment criterion in the labeling rule base is also updated simultaneously.

[0042] For video data, the annotation agent first performs automatic frame extraction and deduplication preprocessing, and then performs the above annotation process on keyframes. When the object to be annotated undergoes significant deformation, occlusion, or rapid movement in the video, the annotation agent uses an optical flow tracing algorithm to assist in generating annotation candidates for non-keyframes after annotating the keyframes. The mapping method between optical flow tracing and the annotation box is as follows: extract the FAST corner points within the annotation box as feature points, calculate the optical flow vectors of each feature point between adjacent frames, take the median of the optical flow vectors of all feature points as the overall displacement of the annotation box, and estimate the scale factor of the annotation box based on the standard deviation of the feature point distribution. To suppress error accumulation in long-sequence tracking, optical flow tracing is reset every N frames based on the keyframes. N is dynamically determined by the annotation agent based on the video frame rate (the higher the frame rate, the larger N) and the intensity of motion (the more intense the motion, the smaller N). For example, for a 30fps video, the value of N ranges from 10 to 20 frames. Optical flow tracking confidence is calculated based on the consistency metric of optical flow vectors, specifically the average angular deviation between the optical flow vectors of each feature point and the median vector. The smaller the deviation, the higher the confidence. Frames with optical flow tracking confidence below a threshold are pushed to the user for confirmation, rather than simply using temporal smoothing interpolation, to avoid annotation bias in complex motion scenarios.

[0043] The entire annotation process is executed by the intelligent agent, which avoids the problem of inconsistent annotation standards among different personnel. All annotation rules and confirmation records are synchronously stored in the interaction context memory pool for reuse in subsequent optimization stages.

[0044] Taking the construction worker safety helmet detection scenario as an example, after a user uploads construction site monitoring video data, they input the natural language requirement "label all workers in the image who are not wearing safety helmets." The labeling agent first parses the requirement and calculates that the basic dimension ambiguity is low, but the quantization threshold missing degree in scene boundary ambiguity is high, and the implicit hypothesis bias does not clearly define the judgment rules for distant workers and back-end management personnel. The overall ambiguity exceeds the threshold, so the agent proactively asks the user, "How many meters away from the camera can workers not be labeled? Should back-end management personnel be included in the labeling range?" After the user replies, "Workers more than 20 meters away do not need to be labeled, and management personnel do not need to be labeled," the labeling agent updates the rules and recalculates the ambiguity. After confirming that it is below the threshold, it generates structured labeling rules. Subsequently, the labeling agent extracts samples that are suitable for the task complexity for the user to label. After the user completes the labeling, the agent completes the pre-labeling of the remaining video frames based on small sample inference and pushes low-confidence samples from distant and edge areas to the user for confirmation. After the user corrects individual erroneous labels using natural language, a complete safety helmet detection labeling dataset is generated. The user inputs training requirements through a natural language dialogue interface. The training agent first retrieves the confirmed annotation rules and task objectives from the interactive context memory pool, eliminating the need for the user to repeat the description. The training agent then further analyzes the training requirements, extracting key information such as accuracy targets and deployment scenarios, while also reading attribute information from the labeled dataset, including data volume, category distribution, and image resolution. Subsequently, the training agent performs model matching from a pre-set basic model library, matching the corresponding detection or classification architecture based on the task type, and selecting an appropriate model size considering data volume and edge deployment requirements. After model selection, the training agent automatically configures hyperparameters using a Gaussian process-based Bayesian optimization algorithm. This algorithm uses a Gaussian process as a surrogate model to fit the mapping relationship between hyperparameters and model accuracy, iteratively selecting the next set of evaluated hyperparameters through a data acquisition function, gradually approaching the optimal hyperparameter combination. In this embodiment, the data acquisition function uses the expected boost function, whose expression is: ; In the formula, x is the hyperparameter combination to be evaluated, including parameters such as learning rate, batch size, weight decay coefficient, and number of iterations; y is the model validation accuracy corresponding to the hyperparameter combination x. This represents the best observed model accuracy value. The probability density function of the accuracy y predicted by the Gaussian process surrogate model under hyperparameter x; Given the expected improvement value for hyperparameter combination x, the algorithm selects the hyperparameter combination with the largest expected improvement value for the next training validation. The number of hyperparameter optimization iterations is dynamically configured based on the task data volume and computing resources. For small to medium-sized datasets (sample size less than 5000), the default is 5 iterations; for large-scale datasets, the default is 8-12 iterations. Users can adjust the number of iterations according to their actual computing power. After the hyperparameters are configured, the training agent calls the distributed training engine of the base layer, scheduling multiple computing nodes of the local computing power cluster to perform distributed training.

[0045] During training, the training agent collects metrics such as training set loss and validation set accuracy at fixed intervals, plots the loss and accuracy curves through a visual interface, and proactively pushes the training status to the user in natural language. When an anomaly is detected—specifically, an increase in training loss and a decrease in validation set accuracy exceeding a preset tolerance threshold within five consecutive training steps (this tolerance threshold is dynamically calculated by the training agent based on the current training round and the target accuracy, e.g., higher tolerance in the early stages and lower tolerance in later stages)—the training agent immediately triggers an active warning, explaining the anomaly type to the user in natural language and providing corresponding adjustment suggestions. The user can confirm the adjustments via natural language, and the training agent automatically updates hyperparameters and continues training without requiring manual parameter modification or interruption. The detection of training anomalies uses a sliding window trend judgment method: the window size is set to 10 training steps, and the linear regression slope of the training loss within the window is calculated as the upward trend strength. When the upward trend strength exceeds a preset threshold (for example, a slope greater than 0.05 for three consecutive windows) and the validation set accuracy simultaneously decreases beyond the tolerance threshold, an anomaly warning is triggered. The tolerance threshold is dynamically calculated by the training agent based on the current training round: in the early stage of training (the first 20% of rounds), the tolerance threshold is higher (e.g., the accuracy of the validation set decreases by no more than 5%); in the later stage of training (the last 20% of rounds), the tolerance threshold is lower (e.g., the accuracy of the validation set decreases by no more than 1%), in order to adapt to the different requirements for accuracy stability at different training stages.

[0046] This invention also constructs an automated iterative closed loop of evaluation-guidance-feedback-tuning. After training, the evaluation agent automatically calls the test dataset to comprehensively evaluate the model, calculating indicators such as overall accuracy, precision for each category, recall, and confusion matrix. Simultaneously, it statistically analyzes all erroneous samples, classifies them using a feature clustering algorithm, and summarizes different error types such as missed detections, false positives, and category confusion, along with their corresponding sample characteristics. A structured evaluation report is generated and displayed to the user through a visual interface. Based on the evaluation results, the tuning agent organizes the error classification, error percentage, and corresponding optimization directions into natural language guidance information and proactively pushes it to the user. The user can directly confirm the optimization direction through natural language or supplement specific optimization requirements without needing to analyze the causes of errors themselves.

[0047] After receiving user feedback, the tuning agent identifies the features and recognition biases of the erroneous samples. It then automatically generates targeted data augmentation strategies, including color perturbation, scale transformation, and background fusion, supplementing the training set with augmented samples generated from the original dataset. Once the samples are supplemented, the tuning agent performs incremental fine-tuning using a layer-freezing strategy. This freezes the parameters of the bottom-level feature extraction layer of the base model, updating only the top-level feature fusion layer and classification head parameters. A small initial learning rate is set for short-cycle training, typically completed within 30 minutes to 2 hours (depending on the amount of supplemented samples and computing power). This significantly shortens the optimization time compared to full retraining. After one round of tuning, the evaluation agent re-evaluates the model and pushes the new evaluation results to the user. The user can then confirm the optimization direction and iterate multiple times until the model meets business requirements. All tuning rules and user preferences are synchronously stored in the interaction context memory pool, allowing subsequent similar tasks to directly reuse the optimization strategies.

[0048] Continuing with the aforementioned example of the helmet detection scenario, after the model training is completed, the evaluation agent test finds that blue helmets have a high false negative rate. The optimization agent proactively pushes a message to the user: "Blue helmets have been detected, accounting for 40% of the total errors. This can be optimized by supplementing blue helmet samples and color enhancement. Do you want to perform optimization?" After the user confirms, the optimization agent automatically performs blue channel perturbation enhancement on the original data, generates supplementary samples, and performs incremental fine-tuning. The model optimization is completed in a short time. After the evaluation confirms that the blue helmet recognition accuracy meets the standard, the user can export the model.

[0049] In this embodiment, all data preprocessing, labeling, training, evaluation, and optimization processes are completed within the local computing cluster. Data and model files are not transmitted over the public network, ensuring data remains within the local domain. The system also features dataset version management, employing incremental storage to automatically record every modification, sample addition, and label adjustment to the dataset, storing only the differences from the previous version. This supports backtracking and comparison of any historical version, ensuring data traceability while controlling storage costs. The system also includes a fine-grained permission management module, configuring different permissions for different user roles, such as data access, model training, and export, ensuring data and model access security. Once the model accuracy meets business requirements, users can trigger model export via natural language commands or visual operations. The training agent automatically converts the model to a deployment format suitable for edge devices, while simultaneously performing quantization and compression optimization. The exported model can then be directly deployed to edge computing devices to perform inference tasks.

[0050] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. An interactive data annotation and model training method based on large-scale intelligent agents, characterized in that, Includes the following steps: S1. Receive the raw multimodal data uploaded by the user and obtain the initial annotation and training requirements input by the user through natural language. S2. Initial requirements are analyzed through the annotation agent. Based on a pre-constructed four-level ambiguity evaluation index system, the ambiguity membership degree of each dimension is calculated using a fuzzy membership function. A dual-mode weighting mechanism combining pre-set expert weights and dynamic entropy weights is used to calculate the overall requirement ambiguity. For scenarios with historical task data or similar reference datasets, the entropy weight method is used to dynamically calculate the weights of each index. For cold start scenarios, pre-set expert weights are used. When the ambiguity exceeds a preset threshold, a hierarchical clarification interaction is initiated according to the level of the ambiguity type and its logical dependencies, in the order of "basic dimension - cross-dimensional association - scenario boundary - implicit hypothesis". After confirmation, structured annotation rules are generated. Data annotation is completed based on a few-shot learning mechanism and multiple rounds of interactive confirmation, generating an annotated dataset. S3. By training the intelligent agent, the training requirements are analyzed, the basic model and hyperparameter configuration are automatically matched, local computing resources are scheduled to execute model training, and the training status and abnormal warnings are proactively fed back to the user during the training process. S4. Evaluate the performance of the trained model by evaluating the agent, classify and count error samples, and generate evaluation results; S5. The optimization agent proactively pushes error classification and optimization direction guidance to the user based on the evaluation results. After receiving optimization feedback from the user through natural language input, it performs targeted data augmentation and incremental fine-tuning to complete the model iteration. S6. Export the model file that meets the performance requirements to the target device.

2. The interactive data annotation and model training method based on large model agents according to claim 1, characterized in that, In step S2, the calculation of the overall demand ambiguity specifically includes: Construct a four-level ambiguity evaluation index system: the target level is the comprehensive demand ambiguity degree D; Criterion layer is the basis for dimensional ambiguity Cross-dimensional association ambiguity Ambiguity of scene boundaries Implicit assumption bias Each criterion layer has a corresponding indicator layer. The ambiguous membership degree of each individual indicator is calculated using a pre-set fuzzy membership function. Specifically, a descending half-trapezoidal membership function is used for confidence level indicators, while an ascending half-trapezoidal membership function is used for positive indicators such as rule conflict degree and missing degree. A dual-mode weighting mechanism combining pre-defined expert weights and dynamic entropy weighting is adopted: For scenarios with historical task data or similar reference datasets, the entropy weight method is used to dynamically calculate the weight of each indicator, and the entropy value of each indicator is calculated through information entropy to obtain the objective weight of each indicator; for cold start scenarios, a criterion layer weight based on domain expert experience is adopted. and indicator layer weights By performing weighted fusion at each level, the overall ambiguity of demand is obtained; Simultaneously, the ambiguity contribution of each indicator is calculated, and the clarification order is determined according to the logical dependency relationship based on the level to which the ambiguity type belongs. Basic dimension ambiguity is dealt with first, followed by cross-dimensional correlation ambiguity, scenario boundary ambiguity, and implicit assumption bias.

3. The interactive data annotation and model training method based on large model agents according to claim 2, characterized in that, After the labeling agent guides the user to complete the initial sample labeling that is adapted to the task complexity, it completes the batch pre-labeling based on the small sample inference capability of the edge lightweight multimodal base model. During the pre-annotation process, boundary samples with confidence levels below the system's preset confidence threshold are identified and actively pushed to the user for confirmation. At the same time, the system receives natural language error correction instructions from the user, corrects the results, and updates the annotation rules and sample example library. The parsing of the natural language error correction instructions includes: locating the error correction object through a referential resolution mechanism, identifying the correction type through error correction intent classification, and mapping the error correction instructions to specific labeled samples for correction through labeled sample association matching.

4. The interactive data annotation and model training method based on large model agents according to claim 1, characterized in that, Step S3 specifically includes: The training agent analyzes the task type, accuracy target, and data attributes in the natural language training requirements, matches the appropriate base model from the preset model library, automatically generates the optimal hyperparameter combination based on the Bayesian optimization algorithm, schedules the local distributed computing power cluster to execute model training, collects training indicators in real time, provides feedback to the user on training progress and convergence status through natural language, and proactively issues warnings and provides adjustment suggestions when training anomalies are detected. The detection of training anomalies uses a sliding window trend judgment: the strength of the upward trend of the training loss within the sliding window is calculated, and an anomaly warning is triggered when the strength of the upward trend exceeds a threshold and the accuracy of the validation set decreases synchronously beyond the tolerance threshold. The tolerance threshold is dynamically calculated by the training agent based on the current training round and the target accuracy of the task.

5. The interactive data annotation and model training method based on large model agents according to claim 1, characterized in that, Step S5 specifically includes: The evaluation agent performs feature clustering and error type classification on erroneous samples, and the tuning agent pushes the error classification and optimization direction to the user in natural language to guide the user to confirm the optimization direction. After receiving user feedback, the model identification deviation is located, corresponding data augmentation strategies are generated and training samples are supplemented. The model is optimized by using layer freeze incremental fine-tuning to quickly complete the model iteration.

6. The interactive data annotation and model training method based on large model agents according to claim 1, characterized in that, A full-link interactive context memory pool is constructed to store users' historical annotation standards, judgment boundaries, optimization preferences, and task objectives. Context information is automatically reused in multi-round interactions and cross-stage tasks to calculate the implicit hypothesis bias and avoid users repeatedly describing their needs. The reuse of historical context in the context memory pool must meet similarity judgment conditions: before reusing historical context, the annotation agent first calculates the semantic similarity between the current task requirement and the historical task. When the similarity exceeds a dynamic threshold, the corresponding historical context is reused; otherwise, the current task requirement takes precedence. The dynamic threshold is adaptively adjusted based on the user's historical reuse confirmation rate. The higher the reuse confirmation rate, the lower the threshold, and vice versa. During system initialization, when the context memory pool is empty, the implicit hypothesis bias is handled by maximizing uncertainty, i.e., a value of 1.0 is set to trigger clarification with a conservative strategy.

7. The interactive data annotation and model training method based on large model agents according to claim 1, characterized in that, The entire process of data annotation, model training, evaluation and optimization is executed in the local computing power device cluster. Data and model files do not leave the local environment. At the same time, fine-grained user permission control and full version traceability management of datasets are configured. Dataset version management adopts incremental storage method, only recording the difference data of each modification to control storage costs.

8. The interactive data annotation and model training method based on large model agents according to claim 1, characterized in that, The multimodal data includes image, video, and audio data, and the supported annotation types include object detection annotation, action recognition annotation, semantic segmentation annotation, and anomaly detection annotation.

9. The interactive data annotation and model training method based on large model agents according to claim 4, characterized in that, The Bayesian optimization algorithm uses a Gaussian process as a surrogate model and iteratively selects the optimal hyperparameter combination by improving the expected value acquisition function to adapt to the parameter requirements of the current training task. The number of hyperparameter optimization iterations is dynamically configured according to the amount of task data and computing resources.

10. The interactive data annotation and model training method based on large model agents according to claim 6, characterized in that, The labeling agent, training agent, evaluation agent, and tuning agent are all encapsulated based on a lightweight multimodal base model on the edge. Each agent interacts with data and collaborates on tasks through a local application programming interface, sharing a unified interaction context memory pool.

Images