A Multimodal Data Labeling Method and System Based on Large Model Pre-labeling

Abstract

This invention discloses a multimodal data annotation method and system based on large model pre-annotation. The method includes: S1: receiving the original multimodal data to be annotated and the annotation task definition, and generating a structured task instruction signal; S2: inputting the structured task instruction signal into the multimodal large model to generate a pre-annotation result signal containing preliminary labels and their corresponding confidence scores; S3: scheduling a manual verification task based on the confidence scores in the pre-annotation result signal; S4: performing parameter fine-tuning or prompting optimization on the multimodal large model according to the manual verification signal, and generating a model optimization signal; S5: using the multimodal large model updated by the model optimization signal to pre-annotate new multimodal original data, and integrating the manual verification signal. This invention's multimodal data annotation method and system based on large model pre-annotation can solve the problems of low efficiency, high cost, and difficulty in unifying quality in traditional multimodal data annotation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and data annotation technology, specifically to a multimodal data annotation method and system based on large model pre-annotation. Background Technology

[0002] With the deepening application of artificial intelligence technology in fields such as autonomous driving, medical image analysis, and intelligent content moderation, the demand for large-scale, high-quality multimodal labeled data (such as images, text, audio, video, and combinations thereof) is growing rapidly. Currently, mainstream data labeling methods heavily rely on manual work, requiring labelers to classify, box-shape, plot points, and transcribe massive amounts of raw data according to specifications. This approach has several significant drawbacks: first, it is extremely inefficient and has high labor costs, becoming a bottleneck for the implementation of AI projects; second, the labeling quality is affected by the subjective experience and state of the personnel, making it difficult to maintain consistency, and noisy data can directly affect the performance of downstream AI models; third, for multimodal data, it is necessary to coordinate labelers from different fields to process the data separately, and then perform alignment and fusion in the later stages, making the process complex and prone to errors.

[0003] To alleviate the pressure of manual annotation, some auxiliary annotation tools based on traditional machine learning models have emerged, such as using pre-trained visual models to detect objects and generate initial bounding boxes. However, these methods have significant limitations: First, they are highly task-specific and domain-specific, with poor generalization ability; the model needs to be retrained for different scenarios or tasks, lacking universality. Second, they are usually unimodal, lacking understanding and utilization of cross-modal semantic relationships, and cannot complete complex annotation tasks that require fusion and understanding, such as "locating objects in an image based on descriptions" or "generating summary text for video clips."

[0004] In recent years, large language models and multimodal large models have made groundbreaking progress. Through pre-training on massive amounts of internet data, they possess powerful general knowledge, contextual understanding, and reasoning capabilities, bringing new hope to automated data annotation. Theoretically, large models can be directly instructed to complete annotation tasks. However, direct application in real-world industrial scenarios faces numerous challenges: the output of large models is unpredictable, and their format may not meet downstream training requirements; their generated results may contain "illusions" or factual errors, requiring full manual review and thus losing their auxiliary value; and there is a lack of a systematic framework that seamlessly integrates the capabilities of large models with existing mature annotation processes, quality management systems, and human expert experience. Therefore, how to construct an intelligent annotation system centered on multimodal large models, capable of reliably and efficiently completing complex annotation tasks, and continuously self-optimizing through human interaction has become a pressing technical problem to be solved. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a multimodal data annotation method and system based on large model pre-annotation, to solve the problems of low efficiency, high cost and difficulty in uniform quality of traditional multimodal data annotation. This invention automates the pre-annotation of raw data using a large multimodal model, generating preliminary results with confidence assessments. Subsequently, the system intelligently schedules human intervention based on the confidence level, performing targeted verification and correction only on low-confidence or complex parts. Finally, the human feedback forms a closed loop for iterative optimization of the large model or suggestion strategies, thereby significantly reducing reliance on human intervention while ensuring annotation quality, achieving a synergistic increase in annotation efficiency and model capabilities.

[0006] This invention provides a multimodal data annotation method based on large model pre-annotation, including:

[0007] S1: Receive the original multimodal data to be labeled and the labeling task definition, and generate structured task instruction signals;

[0008] S2: Input the structured task instruction signal into the multimodal large model to generate a pre-labeled result signal containing preliminary labels and their corresponding confidence scores;

[0009] S3: Based on the confidence level in the pre-labeled result signal, schedule the manual verification task, receive the correction information of the preliminary label by the human, and generate the manual verification signal;

[0010] S4: Based on the manual verification signal, fine-tune or prompt the optimization of parameters for the multimodal large model, and generate the model optimization signal;

[0011] S5: Use the multimodal large model updated by the model-optimized signal to pre-label the new multimodal raw data, and integrate the manual verification signal to output the final labeled dataset.

[0012] In one embodiment of the present invention, the steps of receiving the multimodal raw data to be labeled and the labeling task definition specifically include: acquiring raw data streams from different sensors, wherein the raw data streams contain at least two or more of image sequences, text descriptions, and audio waveforms; performing keyframe extraction and enhancement processing on the image sequences to generate standardized visual signals, performing word segmentation and vectorization processing on the text descriptions to generate text feature signals, and performing frame segmentation and feature extraction on the audio waveforms to generate acoustic feature signals; according to the user-defined labeling specifications, performing time stamp alignment and spatial correlation analysis on the standardized visual signals, text feature signals, and acoustic feature signals, and fusing them to generate a unified multimodal data packet signal; finally, according to the preset labeling task type, combining the multimodal data packet signal with the task requirement template, and encapsulating it to form a structured task instruction signal.

[0013] In one embodiment of the present invention, the structured task instruction signal includes explicit output format specifications and semantic constraints. The output format specifications define the data structure, hierarchical relationship, and mapping method of the labels on different modal data, while the semantic constraints limit the attributes of the labels, the relationship between categories, and the granularity of the annotation through natural language instructions. When the structured task instruction signal is input into a multimodal large model, the instruction parsing module inside the model is first activated. This module decomposes the signal into understanding sub-tasks for different modalities and a unified output generation sub-task. Under the guidance of semantic constraints, the model performs joint inference and cross-modal semantic alignment analysis on the multimodal data, and then generates structured preliminary labels according to the output format specifications. A confidence score reflecting the reliability of the generation is calculated for each label, which together constitute the pre-labeled result signal.

[0014] In one embodiment of the present invention, the multimodal large model is a general artificial intelligence model that has been pre-trained on large-scale cross-modal data and has the ability to understand and generate. The step of generating a pre-labeled result signal containing preliminary labels and their corresponding confidence scores is as follows: After receiving the structured task instruction signal, the multimodal large model performs feature encoding on the corresponding modal data by its internal visual encoder, text encoder and audio encoder respectively to generate high-order feature representation signals. These feature representation signals are input into a shared cross-modal attention fusion network for deep interaction and information complementarity to generate a unified scene understanding representation signal. The scene understanding representation signal is then fed to the task-specific decoder head. The decoder head generates preliminary labels of various types, including classification labels, detection boxes, segmentation masks and descriptive text, in parallel according to the output format specification, and uses the calibration module inside the model to attach a confidence score based on its probability distribution or logical consistency to each generated label, thereby outputting the pre-labeled result signal.

[0015] In one embodiment of the present invention, the step of scheduling manual verification tasks based on the confidence level in the pre-labeled result signals includes: setting three confidence level threshold ranges of high, medium, and low, and automatically sorting the pre-labeled result signals; for preliminary labels with confidence levels higher than the first threshold, directly marking them as verified signals and including them in the final result pool; for preliminary labels with confidence levels between the first and second thresholds, packaging them into a light review task package signal and assigning them to junior annotators for quick verification and confirmation; for preliminary labels with confidence levels lower than the second threshold, packaging them together with related multimodal context data into a deep verification task package signal and assigning them to senior or expert annotators for detailed review and correction; receiving manual correction information for preliminary labels means that the system provides an integrated annotation interface that synchronously displays multimodal data, pre-labeled results, and their confidence levels. Any confirmation, modification, deletion, or addition of labels performed by annotators on this interface is captured in real time and encoded into a structured human-computer interaction event stream, which is then integrated to generate a manual verification signal.

[0016] In one embodiment of the present invention, the process of generating a manual verification signal is further refined into a multi-stage quality control process. First, the system pushes the pre-annotation result signal that requires manual processing to the task queue and dynamically allocates tasks according to the annotator's professional field and current workload, generating an initial allocation signal. After the annotator completes the verification on the interactive interface, an original correction signal is generated. This original correction signal enters an automated logical consistency check module to check whether there are contradictions or conflicts between the new label and the existing label, as well as between different modal labels, and generates a consistency check report signal. If a conflict is found, the relevant task and report signal are sent to another annotator for arbitration verification, generating an arbitration confirmation signal. Finally, the conflict-free original correction signal or the arbitration-confirmed signal is bound with metadata such as task allocation, operation log, and timestamp, and packaged to form a complete and traceable manual verification signal.

[0017] In one embodiment of the present invention, the steps of fine-tuning or prompting optimization of parameters of a large multimodal model based on manual verification signals specifically include: establishing an incremental learning dataset, which consists of the original multimodal data that triggers manual correction, its corresponding pre-labeled result signals, and the correct labels in the finally confirmed manual verification signals; when the incremental learning dataset accumulates to a certain scale, the model fine-tuning process is triggered, and the dataset is used to perform supervised incremental training on some or all parameters of the large multimodal model, updating the model weights by calculating the loss function between the model prediction and the manually confirmed labels and backpropagating, generating an updated model parameter file, which constitutes a form of model optimization signal; simultaneously, the system analyzes the patterns and rules of manual correction, automatically optimizes the prompt template library and rule library used to generate structured task instruction signals, forming an updated knowledge base signal, which serves as another form of model optimization signal to guide the subsequent pre-labeling process.

[0018] In one embodiment of the present invention, when the method pre-labels new original data using the updated multimodal large model, it simultaneously loads a model optimization signal, enabling the new model to incorporate the experience and knowledge of previous manual verification during inference. For the newly generated pre-labeling result signal, the system compares and analyzes its confidence distribution with the historical distribution to generate a model performance evolution evaluation signal. This evaluation signal is used to dynamically adjust the confidence threshold range and the parameters of the manual task scheduling strategy. The newly generated manual verification signal is continuously collected and added to the incremental learning dataset to prepare for the next round of model optimization. The entire process causes the accuracy of pre-labeling, the efficiency of manual verification, and the intelligence level of the entire system to increase iteratively, and the quality of the final output labeled dataset also improves round by round.

[0019] This invention also includes a multimodal data annotation system based on large model pre-annotation, comprising:

[0020] The data input and task definition module is used to receive multimodal raw data and user configuration, and generate structured task instruction signals;

[0021] The large model service and scheduling module integrates or connects one or more multimodal large model instances, and is responsible for receiving structured task instruction signals, calling the corresponding model for calculation, and returning pre-labeled result signals.

[0022] The human-machine collaborative annotation platform module provides an interactive interface, enables intelligent task distribution based on confidence level, and receives human input to generate human verification signals;

[0023] The model optimization and feedback learning module receives manual verification signals, manages incremental learning datasets, performs fine-tuning training of the model or prompts updates to the knowledge base, and generates and stores model optimization signals.

[0024] The annotation process management and output module controls the timing and logic of the entire annotation process, coordinates the signal transmission between modules, and is responsible for integrating pre-annotation results and manual verification results to ultimately output a high-quality annotated dataset.

[0025] In one embodiment of the present invention, the human-machine collaborative annotation platform module further includes an intelligent task scheduling engine and an interactive annotation workbench. The intelligent task scheduling engine continuously monitors the pre-annotation result signals from the large model service and scheduling module. It has an embedded reputation analyzer and resource manager. The reputation analyzer classifies and sorts the confidence levels according to a preset strategy. The resource manager monitors the status, expertise, and work efficiency of all online annotators in real time. The scheduling engine integrates the confidence level and resource status to dynamically allocate annotation tasks to the most suitable annotators or annotator groups in real time, generating an optimal task allocation signal. The interactive annotation workbench adaptively renders multimodal data views and annotation toolsets according to the task type, and receives the task allocation signals pushed by the intelligent task scheduling engine to display the content to be verified to the annotators. All operations of the annotators are captured by the workbench and fed back to the intelligent task scheduling engine in real time for recording and subsequent processing.

[0026] This invention provides a multimodal data annotation method and system based on large model pre-annotation. The method automatically pre-annotates the original data using a large multimodal model to generate preliminary results with confidence assessment. Subsequently, the system intelligently schedules human intervention based on the confidence level, performing targeted verification and correction only on low-confidence or complex parts. Finally, the human feedback forms a closed loop for iterative optimization of the large model or suggestion strategy, thereby significantly reducing reliance on human intervention while ensuring annotation quality, and achieving synergistic growth in annotation efficiency and model capabilities. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of a multimodal data annotation method based on large model pre-annotation;

[0029] Figure 2 This is a system architecture diagram of a multimodal data annotation system based on large model pre-annotation. Detailed Implementation

[0030] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0031] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0032] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0033] Please see Figure 1-2 The image shows a multimodal data annotation method and system based on large model pre-annotation according to the present invention. The multimodal data annotation method based on large model pre-annotation according to the present invention includes: S1: receiving the original multimodal data to be annotated and the annotation task definition, and generating a structured task instruction signal; S2: inputting the structured task instruction signal into the multimodal large model to generate a pre-annotation result signal containing preliminary labels and their corresponding confidence levels; S3: scheduling a manual verification task based on the confidence levels in the pre-annotation result signal, and receiving manual correction information for the preliminary labels to generate a manual verification signal; S4: performing parameter fine-tuning or prompt optimization on the multimodal large model according to the manual verification signal to generate a model optimization signal; S5: using the multimodal large model updated by the model optimization signal to pre-annotate the new multimodal original data, and fusing the manual verification signal to output the final annotated dataset.

[0034] like Figure 1As shown, the core process of the method provided by this invention begins with the reception and parsing of heterogeneous raw data. Specifically, the system simultaneously receives data streams from multiple sources, which may include, but are not limited to, continuous image frames captured by a camera, language descriptions provided by text files or speech-to-text transcription, and audio waveforms recorded by a microphone array. These raw data are often inconsistent in format, size, and timing. Therefore, the method first performs a series of standardized preprocessing operations: for image sequences, keyframe extraction may be performed to reduce redundancy, and enhancement processing such as size normalization and illumination correction may be performed to transform them into a series of regular visual signal units; for text descriptions, word segmentation, stop word removal, and transformation into machine-understandable word vectors or embedded representations are performed to form text feature signals; for audio waveforms, acoustic feature signals characterizing sound properties are generated by framing, windowing, and extraction of acoustic features such as Mel-frequency cepstral coefficients. These pre-processed single-modal signals do not exist in isolation. The system performs precise alignment and correlation analysis based on their inherent timestamp information or event correlations. For example, it binds a speech description at a certain moment with a specific visual object appearing at the same time, ultimately packaging them into a spatiotemporally or semantically unified multimodal data packet. Simultaneously, users define the specific requirements for this annotation through the system interface, such as object detection in images, sentiment classification in audio, or cross-modal image-text matching. The system has pre-set various task templates; combining these user definitions with the templates generates a clear and unambiguous structured task instruction signal. This instruction signal not only indicates "what to do" but also implicitly contains preliminary specifications for "how to output." The core of this method is the use of large-scale pre-trained multimodal intelligent models for automated pre-annotation. The generated structured task instruction signal is transmitted to one or more large multimodal model servers. These large models are typically trained on massive amounts of image-text pairs, video-audio-text triples, and other data, possessing powerful cross-modal understanding and generation capabilities. Once the model receives instructions, its complex internal neural network architecture begins to work collaboratively: the visual encoder parses image information, the text encoder understands language instructions and text descriptions, and the audio encoder processes sound features. These encoders transform the raw data into a high-dimensional feature representation. More importantly, the model's internal cross-modal attention fusion mechanism actively establishes connections between features from these different sources, engaging in deep information interaction and complementarity. For example, it uses keywords from text descriptions to enhance the visual features of relevant regions in an image, or it uses the content of the image to aid in understanding ambiguous speech fragments, thus forming a comprehensive and unified representation for scene understanding. Based on this fused deep representation, the decoder head corresponding to the task in the model begins to function.If the task is directive like "label all the cars in the image," the decoder will output a series of bounding box coordinates and a "car" label. If the task is "describe the main content of the image," the decoder will generate a coherent text. When generating each initial label (whether coordinates, category, or text), the model internally calculates a confidence score based on probability distribution or logical reasoning. This score quantifies the model's confidence in the output. All these initial labels and their accompanying confidence scores are encapsulated together to form a pre-labeled result signal, which is then returned to the system. This step, which automates a large amount of repetitive and rule-based labeling work, is fundamental to improving efficiency.

[0035] Furthermore, due to the inherent "illusion" tendency of large models and the uncertainty in handling complex and marginal cases, relying entirely on their automatic output is unreliable. Therefore, this invention introduces an intelligent manual verification scheduling mechanism based on confidence levels. Upon receiving the pre-labeled result signal, the system immediately analyzes the confidence scores. Based on a pre-set strategy (e.g., setting high, medium, and low threshold ranges), the system automatically sorts the results. For results with the highest confidence scores, the system can directly mark them as high-confidence results and include them in the final result set without manual intervention, handling most simple and straightforward cases. For results with confidence scores in the middle range, indicating that the model has some confidence but uncertainty exists, the system packages them into a light review task and assigns them to junior or ordinary annotators for quick review and confirmation; they may only need to click "confirm" or make minor adjustments. For results with the lowest confidence scores, it means the model is "confused" or lacks sufficient basis for judgment; these are often the difficulties and key points in annotation tasks. The system packages these results, along with their rich multimodal contextual data, into deep validation tasks, which are then specifically assigned to experienced senior or expert annotators. To achieve this scheduling, the system provides a highly integrated interactive annotation interface. On this interface, annotators can simultaneously see images, hear audio, and read text, and intuitively view the pre-annotated bounding boxes, labels, or descriptive text of the model, as well as confidence levels clearly indicated by color or numerical values. Annotators can perform various operations on this interface: confirming correct labels, dragging to correct inaccurate detection boxes, modifying incorrect classifications, deleting redundant annotations, or supplementing missing content in the model. All these interactive operations are captured by the system in real time and accurately, and transformed into structured correction instructions. These instructions are combined with metadata such as task identifiers and operator information to ultimately generate a clear and traceable human validation signal. This step cleverly optimizes the allocation of human resources, precisely directing valuable human attention to where it is most needed.

[0036] In one embodiment of the invention, the method constructs a continuously self-improving feedback optimization closed loop. The value of human verification signals goes far beyond generating a batch of correctly labeled data. The system specifically collects samples that triggered human corrections—that is, the areas where the model initially predicted incorrectly or was uncertain. It then maps these samples' original data, the model's incorrect pre-labels, and the correctly provided human labels to form a high-quality incremental learning dataset. When such data accumulates to a certain scale, it can trigger a fine-tuning process for the multimodal large model. Specifically, this incremental dataset is used to perform supervised retraining on some or all of the model's parameters. Optimization algorithms are used to make the model more likely to produce outputs consistent with the human corrections when encountering similar situations in the future. This process generates a new model parameter file, constituting a substantial model optimization signal used to update the model version in the production environment. Besides directly adjusting model parameters, another lighter optimization approach is to extract the expert experience reflected in human verification. The system can analyze patterns in a large number of correction examples, such as discovering that the model consistently confuses two specific object categories, or that the labeling format is prone to errors in certain specific scenarios. These patterns can be formalized into more precise prompts, more detailed rules, or more comprehensive output format templates, thereby updating the knowledge base of the system-generated "structured task instruction signals." This optimization generates a strategic model optimization signal. Whether it's parameter updates or strategy updates, their effects are reflected in the next round of pre-labeling of new data: the optimized model or strategy can produce higher-quality pre-labeled results with better confidence distributions, thereby reducing the amount of manual verification required in the next round and potentially increasing the difficulty level of the focus task for manual verification. This cycle repeats, forming a continuously reinforcing loop of "labeling-verification-optimization-relabeling." Through this closed loop, the entire system not only completes the one-time labeling project but also continuously evolves its core intelligent labeling capabilities as it is used, resulting in a step-by-step improvement in the quality of the final labeled dataset and the efficiency of the entire process. Through the organic combination of the above four core steps, this invention constructs an efficient, accurate, and learning-adaptive intelligent multimodal data labeling solution.

[0037] like Figure 1As shown, a more refined and rigorous quality control design was implemented for the generation process of manual verification signals, expanding the single manual operation into a multi-stage process that includes parallel verification and conflict resolution mechanisms. This process begins after the task allocation signal is generated. Instead of simply pushing the task to be verified directly to a single annotator, the system introduces a dynamic task routing and load balancing mechanism. This mechanism monitors in real time the online status of all available annotators, their historical annotation accuracy, their proficiency in different task types, and the current number of pending tasks. It calculates the optimal allocation scheme for each new task based on these factors, generating an initial task assignment signal. When an annotator completes modifications and submits them on the interactive interface, the resulting original correction signal is not immediately adopted as the final result. The system initiates a background automated logical consistency check module, which scans the corrected result based on predefined domain knowledge rules and common sense. For example, in autonomous driving scenarios, the system checks whether an object labeled "car" has a reasonable size, or in an indoor scene, it checks for illogical labels like "sky." For multimodal data, it checks for significant discrepancies between text and image descriptions, or for large temporal deviations between audio transcription and speaker lip movements. Once a potential contradiction or conflict is detected, the module automatically generates a detailed consistency check report signal, indicating the specific location and nature of the conflict. Subsequently, the system encapsulates the task, the original correction signal, and the check report signal together and sends them to another senior annotator or a dedicated quality control officer for arbitration verification. The arbitrator needs to review the conflict points, determine whether the initial correction is reasonable, or propose a third solution. The arbitration confirmation signal generated after the arbitrator's confirmation has higher authority. Finally, the system deeply binds the original correction signals that have passed the consistency check, or the final signals confirmed by arbitration, with complete contextual information (including the original task assignment signal, operation logs of all participants, timestamps of each step, and the final arbitration conclusion), packaging them into a fully traceable, quality control-rich manual verification signal package. This signal packet not only contains the correct label data, but also records the decision-making path and quality evidence for achieving this correct result, providing extremely valuable data for subsequent model optimization and process auditing.

[0038] Furthermore, the specific implementation path of the model optimization and feedback learning mechanism is elaborated, clarifying how to effectively transform manual verification signals into a driving force for improving the core intelligence capabilities of the system. The core of this mechanism is the construction and management of a structured incremental learning dataset. The system establishes a dedicated data storage and management unit that continuously receives manual verification signals from the human-machine collaborative process. This unit does not simply accumulate data but performs a series of preprocessing operations: first, it cleans the data, removing invalid records caused by erroneous operations; second, it performs deduplication analysis to avoid duplicate inclusion of identical correction examples, which would affect training efficiency; finally, it structures the data according to dimensions such as task type, data modality, and correction category, and adds rich metadata tags to it, thus forming an incremental learning dataset that is easy to retrieve and use. When the size of this dataset reaches the preset triggering conditions (e.g., accumulating a certain number or covering a sufficient number of error types), the system will automatically or, after administrator confirmation, initiate the model fine-tuning process. During fine-tuning, the model training unit obtains the basic parameters of the model used in the current production environment from the large model service module as a starting point. Then, supervised incremental training is performed using a carefully prepared incremental learning dataset. The essence of training is to allow the model to learn the mapping from "wrong" to "correct": by calculating the difference (loss function) between the model's predictions based on old parameters (i.e., its initial incorrect or uncertain output) and the manually provided standard answers in the dataset, and using backpropagation to update the model's neural network weights. After multiple iterations of optimization, the model parameters are adjusted to significantly increase the probability of generating correct answers when facing similar scenarios in the future. The new parameter file generated after training constitutes the most direct and core model optimization signal, which can be used to update the online model, achieving immediate capability enhancement. Simultaneously, a parallel knowledge extraction unit is also running. This unit uses data analysis and pattern recognition techniques to perform macro-analysis of the manual correction behaviors in the incremental learning dataset. It attempts to discover the common patterns and expert experience hidden behind a large number of specific corrections; for example, the model may easily miss a certain type of object under certain lighting conditions, or frequently make incorrect entity relationship judgments for certain sentence structures. These patterns are formally extracted and transformed into more precise system prompt word templates, more detailed post-processing rules, or more comprehensive output format constraints. This extracted knowledge is updated in the knowledge base of the task definition and instruction generation module at the system's front end, forming a strategic and guiding model optimization signal. When generating structured task instructions next time, this new knowledge will be applied, thereby guiding the model to produce more standardized output from the input instruction level. Parameter fine-tuning and prompt optimization often work together; the former changes the model's "thinking habits" from the bottom up, while the latter optimizes the model's "communication methods" from the top up, jointly driving the steady improvement of the system's intelligence level.

[0039] Specifically, the paper emphasizes the dynamic and adaptive closed-loop system characteristics of the entire method, explaining how the system achieves continuous co-evolution of annotation quality, efficiency, and model performance through the circulation and feedback of internal signals. The wisdom of this closed loop lies in the fact that it is not merely a linear process of "using the model - manual correction - optimizing the model," but an organic whole where each link influences the others and parameters are dynamically adjusted. When the optimized multimodal model is applied to the pre-labeling of a new batch of data, it carries the lessons learned from the previous round of manual verification. Therefore, the inherent confidence score distribution of the newly generated pre-labeling results will change: the overall confidence level may increase, the proportion of high-confidence results may increase, or the confidence in certain previously error-prone categories may become more accurate. The system continuously monitors and analyzes the historical trend of this confidence distribution, generating a model performance evolution evaluation signal. This evaluation signal has significant guiding value; it is fed back to the manual verification scheduling module to dynamically adjust the confidence level threshold range. For example, if the system finds that the overall confidence level of the model significantly and steadily improves on a certain type of task, it can automatically raise the "direct pass" threshold, allowing more results to avoid manual review, thereby further improving throughput while ensuring quality. Conversely, if the model is found to be unstable in newly added data domains, the threshold can be temporarily lowered, introducing more stringent manual review. Simultaneously, this evaluation signal can also be used to optimize the task scheduling strategy itself, such as adjusting the weight ratio of tasks assigned to different levels of annotators for different confidence intervals. On the other hand, newly generated manual verification signals, especially corrections for samples where the optimized model still makes mistakes or is uncertain, are collected in real time and continuously fed into the aforementioned incremental learning dataset. This means that the incremental learning dataset itself is also constantly evolving, always focusing on the system's weakest and most cutting-edge challenges, thus driving the next round of model optimization to be more targeted. In this way, the entire system forms a powerful positive feedback loop: high-quality pre-labeling reduces unnecessary manual labor, high-quality manual verification generates high-quality feedback data, high-quality feedback data drives more accurate model optimization, and the optimized model produces even higher-quality pre-labeling. In this cycle, the overall efficiency of the annotation process (reflected in the optimal configuration of human-machine collaboration), the quality of the final output dataset (reflected in accuracy and consistency), and the domain adaptability of the core large model are no longer mutually exclusive or even contradictory goals. Instead, they have become an organic whole that grows synergistically and promotes each other. The system, like an organism with learning vitality, continuously accumulates experience in actual annotation tasks, self-adjusts, becomes more intelligent with use, and ultimately achieves sustainable improvement in efficiency and effectiveness.

[0040] like Figure 2As shown, this invention provides a system for multimodal data annotation based on large model pre-annotation, comprising: a data input and task definition module, used to receive multimodal raw data and user configuration, and generate structured task instruction signals; a large model service and scheduling module, which integrates or connects one or more multimodal large model instances, is responsible for receiving structured task instruction signals and calling the corresponding model for calculation, and returning pre-annotation result signals; a human-computer collaborative annotation platform module, which provides an interactive interface, realizes intelligent task distribution based on confidence, receives human input, and generates human verification signals; a model optimization and feedback learning module, which receives human verification signals, manages incremental learning datasets, performs fine-tuning training of models or prompts knowledge base updates, and generates and stores model optimization signals; and an annotation process management and output module, which controls the timing and logic of the entire annotation process, coordinates signal transmission between modules, and is responsible for integrating pre-annotation results and human verification results, and finally outputs a high-quality annotated dataset.

[0041] The system architecture constructed in this invention is a modular, highly cohesive, and loosely coupled software system. Its core design lies in decomposing the complex intelligent annotation process into a series of clearly defined functional modules that communicate through standardized signal interfaces. The entire system's operation begins with the data input and task definition module, which acts as the system's "sensory and planning hub." It provides a user-friendly graphical user interface or application programming interface, allowing users or upstream systems to batch import multimodal raw data such as images, videos, text files, and audio streams. This module embeds a powerful data parser and preprocessor, capable of automatically recognizing common data formats and performing the standardized preprocessing operations described above, transforming the raw data into an internally unified intermediate representation. Simultaneously, users configure specific details of the annotation task through this module, such as selecting the annotation type, defining the label system, and setting output format specifications. The core function of this module is to deeply integrate and logically encapsulate the preprocessed multimodal data packets with the user-configured task rules, generating a structured task instruction signal that can be accurately parsed by the machine. This signal serves as the blueprint for all subsequent automated operations, ensuring the unambiguous transmission of task intent.

[0042] Furthermore, this structured task instruction signal is then passed to the large model service and scheduling module, which is the "intelligent computing core" of the system. It is not simply an encapsulation of a single large model, but rather designed as a service gateway capable of managing multiple model instances. This module maintains a model registry, allowing access to multimodal large models from different vendors with varying capabilities and focuses. Upon receiving the task instruction signal, it automatically selects the most suitable model based on the characteristics of the task content, or, in some complex scenarios, decomposes the task and schedules different models to complete it collaboratively. This module handles all the technical details of interacting with the large model, including converting internal signals into prompts that the model can understand, managing the queue and timeouts of call requests, processing the model's response, and parsing it back into a standardized format within the system. When the large model returns results containing preliminary labels and confidence levels, this module performs preliminary encapsulation and verification, generates a uniformly formatted pre-labeled result signal, and publishes it to the system's message bus or directly passes it to downstream modules. The human-machine collaborative annotation platform module, acting as the system's "interaction and scheduling hub," then begins operation. This module consists of two closely collaborating parts: a backend intelligent scheduling engine and a frontend interactive annotation workbench. The intelligent scheduling engine continuously monitors pre-labeling result signals from upstream. Its built-in strategy engine analyzes the confidence distribution of each result in real time and, combined with predefined business rules, classifies and prioritizes the massive amount of pre-labeled results. Simultaneously, the scheduling engine monitors the work status, skill tags, historical quality data, and current workload of all annotators in real time through the resource manager. Based on the principle of "assigning the most suitable task to the most suitable person," the scheduling engine performs millisecond-level matching calculations and dynamically generates task allocation instructions. The interactive annotation workbench is the sole entry point for annotators to interact with the system; it is a highly dynamic web application. Once annotators log in, the workbench retrieves the tasks assigned to them from the scheduling engine in real time. For different task types, the workbench automatically loads corresponding visualization components and annotation tools. For example, for image object detection tasks, it renders the image and loads the frame tool; for text sentiment analysis, it highlights the paragraphs to be analyzed and provides sentiment option buttons. The workbench clearly overlays the model pre-labeling results onto the original data and prominently displays the confidence level. All confirmation, modification, and rejection operations by the labelers are captured in real time and generated into structured logs. These logs are fed back to the scheduling engine and converge to form the final manual verification signal.

[0043] Specifically, the key to the system's continuous evolution lies in the model optimization and feedback learning module, which serves as the "brain, learning, and memory center" of the entire system. This module processes human verification signals from the human-machine collaborative platform in a pipeline manner. First, its incremental data management unit acts like a sophisticated warehouse, responsible for receiving these signals. It doesn't accept all signals but performs rigorous incoming quality checks: including data deduplication, format validation, and correlation checks, ensuring that only high-quality, effective correction data is stored. This data is indexed and stored according to dimensions such as task type, model version, and error mode, forming a structured incremental learning dataset. When the dataset accumulates to a trigger threshold, the model training unit is activated. It obtains a copy and weights of the current baseline model from the large model service module and uses the incremental learning dataset to perform supervised fine-tuning training. The entire training process may be completed on a dedicated high-performance computing cluster, and a new model parameter file is produced after training. Following this, the knowledge extraction unit operates in parallel, using data mining techniques to analyze patterns in the incremental dataset. For example, it summarizes the rule that "when there is certain texture interference in an image, the model is prone to misclassifying object A as object B," and formalizes these rules into "if-then" rules or optimized prompt templates. Both new model parameter files and newly extracted rules and templates are collectively referred to as model optimization signals, which are pushed back to the large model service module and the data input and task definition module, respectively, to complete the real-time updates and enhancements to the system's core capabilities. Finally, the "central nervous system" ensuring the orderly and collaborative work of all the above modules is the annotation process management and output module. This module is a state machine-driven process controller that defines the complete workflow status and transition conditions from task creation, data preprocessing, model inference, manual verification, to model optimization and final output. It monitors the real-time status of each task, coordinates the signal transmission order and dependencies between modules, and handles abnormal situations such as task failure retry and timeout transfer. At the end of the process, this module is responsible for aggregating all validated annotation results, packaging them according to the user-required data format, encrypting them, or uploading them to the designated storage location to generate a deliverable, high-quality annotation dataset. Simultaneously, this module provides a global management dashboard view, displaying key performance indicators (KPIs) for the entire process to administrators, such as task throughput, average processing time, model confidence trends, and human intervention rate, providing data support for process optimization.

[0044] In one embodiment of the invention, the focus is on the internal workings of the human-machine collaborative annotation platform module, where its intelligent task scheduling engine is a complex decision-making system. Internally, it runs a real-time scoring and matching algorithm. For each incoming task to be verified, the algorithm considers not only its confidence score but also analyzes the task's implicit characteristics, such as the complexity of the data domain and the required professional knowledge domain for annotation, generating a multi-dimensional feature vector for the task. Simultaneously, for each annotator, the system maintains a dynamically updated capability profile, including their proficiency in various tasks, average annotation speed, historical accuracy, and real-time workload. The core task of the scheduling engine is to optimally match the task feature vector with the annotator's capability profile within milliseconds. This matching is not simply "assigning the lowest confidence task to the highest-level annotator," but rather based on considerations of global efficiency and quality optimization. It might package a series of similar low-confidence tasks and distribute them to an expert to improve processing consistency, or distribute high-confidence, simple tasks to novices for training and review. The allocation signal generated by this engine is dynamic and adaptive, automatically adjusting the strategy according to the real-time queue length and the number of available personnel. The interactive annotation workbench, seamlessly integrated with this engine, is a highly context-aware client application. Upon receiving task allocation signals from the scheduling engine, it dynamically loads an operating environment tailored to that task. This environment goes beyond simply arranging data and tools; it intelligently organizes the context. For example, for a task requiring the annotation of the motion trajectories of multiple objects in a video, the workbench might simultaneously display the trajectory curves of all annotated objects in the sidebar while the video plays on the main screen, highlighting the object to be processed in the current frame. The workbench presents the results of model pre-annotation in a non-intrusive yet clear manner: high-confidence correct suggestions might be displayed with a semi-transparent blue box, while low-confidence suggestions or those potentially conflicting with the context are indicated with a flashing red box. Annotators can operate efficiently using voice commands, keyboard shortcuts, drag-and-drop, and other methods. Each action is recorded by the workbench as an event with precise timestamps and semantics. These event streams are sent back to the scheduling engine in real time, not only to generate manual verification signals but also to update the annotator's competency profile, enabling continuous optimization of the scheduling strategy. Delving deeper into the model optimization and feedback learning module, its incremental data management unit acts as a "data alchemist." The raw, manually verified signals it receives are numerous and complex. This unit first executes cleaning rules to filter out invalid records such as short-lived operations caused by accidental touches or undo operations. Then, it performs correlation and deduplication, merging multiple corrections for the same data object into the final version and preventing identical correction samples from being repeatedly entered into the database. More importantly, it assigns rich contextual tags to each valid data entry, such as: the original model version number that triggered the correction, the task type, the error category, the complexity of the correction operation, and the level of the annotator involved in the correction.This metadata enables highly targeted sampling and learning for subsequent model training and knowledge extraction. The model training unit initiates when trigger conditions are met. It may employ various fine-tuning strategies; for example, for large models where full parameter fine-tuning is costly, it may use more efficient adapter or prefix tuning techniques, training only a small number of newly added parameters, significantly reducing computational overhead while maintaining performance. The training process involves multiple iterations, monitoring performance on a reserved validation set to prevent overfitting to specific corrected samples. The final generated new model parameter file undergoes a rigorous automated testing process for evaluation; only versions that meet performance standards and do not compromise existing capabilities are approved as official model optimization signals.

[0045] This invention discloses a multimodal data annotation method and system based on large model pre-annotation. The method automatically pre-annotates raw data using a large multimodal model, generating preliminary results with confidence assessments. Subsequently, the system intelligently schedules human intervention based on confidence levels, performing targeted verification and correction only on low-confidence or complex parts. Finally, the human feedback forms a closed loop for iterative optimization of the large model or suggestion strategies. This significantly reduces reliance on human intervention while ensuring annotation quality, achieving synergistic growth in annotation efficiency and model capabilities.

[0046] Therefore, the multimodal data annotation method and system based on large model pre-annotation of the present invention can solve the problems of low efficiency, high cost and difficulty in unifying quality of traditional multimodal data annotation.

[0047] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A multimodal data annotation method based on large model pre-annotation, characterized in that, include: S1: Receive the original multimodal data to be labeled and the labeling task definition, and generate structured task instruction signals; S2: The structured task instruction signal is input into the multimodal large model to generate a pre-labeled result signal containing preliminary labels and their corresponding confidence scores. The structured task instruction signal contains explicit output format specifications and semantic constraints. The output format specifications define the data structure, hierarchical relationship, and mapping method of the labels on different modal data. The semantic constraints limit the attributes of the labels, the relationship between categories, and the granularity of the annotation through natural language instructions. When the structured task instruction signal is input into the multimodal large model, the instruction parsing module inside the model is first activated. This module decomposes the signal into understanding sub-tasks for different modalities and a unified output generation sub-task. Under the guidance of the semantic constraints, the model performs joint inference and cross-modal semantic alignment analysis on the multimodal data, and then generates structured preliminary labels according to the output format specifications. A confidence score reflecting the reliability of the generation is calculated for each label, which together constitute the pre-labeled result signal. S3: Based on the confidence level in the pre-labeled result signal, schedule the manual verification task, receive the manual correction information of the initial label, and generate the manual verification signal; The steps for scheduling manual verification tasks based on the confidence level of the pre-labeled result signals include: setting three confidence level threshold ranges (high, medium, and low) and automatically sorting the pre-labeled result signals; for preliminary labels with confidence levels higher than the first threshold, directly marking them as verified signals and including them in the final result pool; for preliminary labels with confidence levels between the first and second thresholds, packaging them into a light review task package signal and assigning it to junior annotators for quick verification and confirmation; for preliminary labels with confidence levels lower than the second threshold, packaging them together with relevant multimodal context data into a deep verification task package signal and assigning it to senior or expert annotators for detailed review and correction; receiving manual correction information for preliminary labels refers to the system providing an integrated annotation interface that synchronously displays multimodal data, pre-labeled results, and their confidence levels. Any confirmation, modification, deletion, or addition of labels performed by annotators on this interface is captured in real time and encoded into a structured human-computer interaction event stream, which is then integrated to generate the manual verification signal; S4: Based on the manual verification signal, fine-tune or optimize the parameters of the multimodal large model to generate a model optimization signal; the multimodal large model is a general artificial intelligence model with understanding and generation capabilities that has been pre-trained on large-scale cross-modal data; the step of generating a pre-labeled result signal containing preliminary labels and their corresponding confidence scores is as follows: after receiving the structured task instruction signal, the multimodal large model's internal visual encoder, text encoder, and audio encoder respectively encode the features of the corresponding modal data to generate high-order feature representation signals; these feature representation signals are input into a shared cross-modal attention fusion network for deep interaction and information complementarity to generate a unified scene understanding representation signal; the scene understanding representation signal is then fed to the task-specific decoder head, which generates preliminary labels of various types, including classification labels, detection boxes, segmentation masks, and descriptive text, in parallel according to the output format specification, and uses the calibration module inside the model to attach a confidence score based on its probability distribution or logical consistency to each generated label, thereby outputting the pre-labeled result signal; S5: Use the multimodal large model updated by the optimized signal to pre-label the new multimodal raw data, and integrate the manual verification signal to output the final labeled dataset.

2. The multimodal data annotation method based on large model pre-annotation according to claim 1, characterized in that, The steps of receiving the multimodal raw data to be labeled and the labeling task definition specifically include: acquiring raw data streams from different sensors, wherein the raw data streams contain at least two or more of image sequences, text descriptions, and audio waveforms; performing keyframe extraction and enhancement processing on the image sequences to generate standardized visual signals, performing word segmentation and vectorization processing on the text descriptions to generate text feature signals, and performing frame segmentation and feature extraction on the audio waveforms to generate acoustic feature signals; according to the user-defined labeling specifications, performing time stamp alignment and spatial correlation analysis on the standardized visual signals, text feature signals, and acoustic feature signals, and fusing them to generate a unified multimodal data packet signal; finally, according to the preset labeling task type, combining the multimodal data packet signal with the task requirement template, and encapsulating it to form the structured task instruction signal.

3. The multimodal data annotation method based on large model pre-annotation according to claim 1, characterized in that, The process of generating manual verification signals is further refined into a multi-stage quality control process. First, the system pushes the pre-annotation result signals that require manual processing to the task queue and dynamically allocates tasks based on the annotator's professional field and current workload, generating an initial allocation signal. After the annotator completes the verification on the interactive interface, an original correction signal is generated. This original correction signal enters an automated logical consistency check module to check whether there are contradictions or conflicts between the new label and the existing label, as well as between different modal labels, and generates a consistency check report signal. If a conflict is found, the relevant task and report signal are sent to another annotator for arbitration verification to generate an arbitration confirmation signal. Finally, the original correction signal without conflict or the signal confirmed by arbitration is bound with the task assignment, operation log, and timestamp metadata to form a complete and traceable manual verification signal.

4. The multimodal data annotation method based on large model pre-annotation according to claim 1, characterized in that, The steps of fine-tuning or prompting optimization of the multimodal large model based on the manual verification signal specifically include: establishing an incremental learning dataset, which consists of the original multimodal data that triggered manual correction, its corresponding pre-labeled result signal, and the correct labels in the finally confirmed manual verification signal; when the incremental learning dataset accumulates to a certain scale, the model fine-tuning process is triggered, and the dataset is used to perform supervised incremental training on some or all parameters of the multimodal large model, updating the model weights by calculating the loss function between the model prediction and the manually confirmed label and backpropagating, generating an updated model parameter file, which constitutes a form of model optimization signal; at the same time, the system analyzes the rules and patterns of manual correction, automatically optimizes the prompt template library and rule library used to generate the structured task instruction signal, forming an updated knowledge base signal, which serves as another form of model optimization signal to guide the subsequent pre-labeling process.

5. A multimodal data annotation method based on large model pre-annotation according to claim 4, characterized in that, When the method pre-labels new raw data using the updated multimodal large model, it simultaneously loads the model optimization signal, enabling the new model to incorporate the experience and knowledge gained from previous manual verification during inference. For the newly generated pre-labeling result signal, the system compares and analyzes its confidence distribution with historical distributions to generate a model performance evolution evaluation signal. This evaluation signal is used to dynamically adjust the parameters of the confidence threshold range and the manual task scheduling strategy. The newly generated manual verification signal is continuously collected and added to the incremental learning dataset to prepare for the next round of model optimization. The entire process causes the accuracy of pre-labeling, the efficiency of manual verification, and the overall intelligence level of the system to increase iteratively, and the quality of the final output labeled dataset also improves round by round.

6. A multimodal data annotation system based on large model pre-annotation, characterized in that, The system is used to implement the multimodal data annotation method based on large model pre-annotation as described in any one of claims 1-5, including: The data input and task definition module is used to receive multimodal raw data and user configuration, and generate the structured task instruction signal; The large model service and scheduling module integrates or connects one or more multimodal large model instances, and is responsible for receiving the structured task instruction signal, calling the corresponding model for calculation, and returning the pre-labeled result signal; The human-machine collaborative annotation platform module provides an interactive interface to realize intelligent task distribution based on confidence level, and receives human input to generate the human verification signal; The model optimization and feedback learning module receives manual verification signals, manages incremental learning datasets, performs fine-tuning training of the model or prompts knowledge base updates, and generates and stores the model optimization signals. The annotation process management and output module controls the timing and logic of the entire annotation process, coordinates the signal transmission between modules, and is responsible for integrating pre-annotation results and manual verification results to ultimately output a high-quality annotated dataset.

7. A multimodal data annotation system based on large model pre-annotation according to claim 6, characterized in that, The human-machine collaborative annotation platform module further includes an intelligent task scheduling engine and an interactive annotation workbench. The intelligent task scheduling engine continuously monitors the pre-annotation result signals from the large model service and scheduling module. It has an embedded reputation analyzer and resource manager. The reputation analyzer classifies and sorts the confidence levels according to a preset strategy, and the resource manager monitors the status, expertise, and work efficiency of all online annotators in real time. The scheduling engine integrates the confidence level and resource status to dynamically allocate annotation tasks to the most suitable annotators or annotator groups in real time, generating an optimal task allocation signal. The interactive annotation workbench adaptively renders multimodal data views and annotation toolsets according to the task type, and receives the task allocation signals pushed by the intelligent task scheduling engine, displaying the content to be verified to the annotators. All operations of the annotators are captured by the workbench and immediately fed back to the intelligent task scheduling engine for recording and subsequent processing.

Images