Interactive intelligent labeling method and system for multi-modal data

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an interactive intelligent labeling method and system for multi-modal data, and the method comprises the following steps: S1, obtaining multi-modal data to be labeled, the multi-modal data to be labeled comprising one or more of image data, text data, audio data, video data, three-dimensional point cloud data and sensor data; and S2, preprocessing the multi-modal data to be labeled, and converting the heterogeneous original multi-modal data into a unified standardized format. According to the method, preliminary intelligent labeling is carried out through the multi-modal fusion model, and an interactive interface is combined to guide manpower to carry out accurate correction on regions with low confidence and inconsistent modals, so that the overall workload of manual labeling is reduced, frequent switching of labeling personnel among different modal operation interfaces is avoided, and the labeling efficiency is improved. The problems that in the prior art, multi-modal data labeling is low in efficiency, high in cost and long in period are solved, and labeling efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an interactive intelligent annotation method and system for multimodal data. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, multimodal data has become one of the core driving forces for technological progress. Multimodal data annotation, as a crucial step in training AI models, primarily involves assigning accurate semantic labels and relationships to information from different modalities, thereby constructing high-quality training datasets. This requires annotators to not only identify features within a single modality but also understand the deep semantic relationships between cross-modal data, ensuring the model can learn and generalize effectively. As deep learning models increasingly demand higher data volumes and annotation quality, efficient and accurate multimodal data annotation methods have become a fundamental guarantee for the practical application of artificial intelligence.

[0003] After searching, the following problems were found in the existing technology: 1. Whether relying on manual outsourcing or traditional open-source tools, the complexity of multimodal data results in a massive workload, high costs, and lengthy processing times for annotation. For example, in autonomous driving scenarios, the collaborative annotation of multi-source data such as images and radar point clouds lacks a unified and efficient mechanism. Annotators need to frequently switch between different operating interfaces, making the process cumbersome and extremely inefficient when dealing with massive amounts of data, failing to meet the rapidly growing demand for AI training data. Furthermore, the lack of effective collaborative annotation mechanisms between different modalities further reduces overall annotation efficiency.

[0004] 2. Due to the inherent semantic complexity of multimodal data, annotators are prone to biases when processing and associating information from different modalities. For example, inconsistencies between text descriptions and image details can lead to incorrect labeling. More seriously, existing annotation tools and methods generally lack effective automatic verification and correction mechanisms, making it difficult to detect and correct errors in a timely manner during the annotation process. Even with simple review functions, it is difficult to handle the complex semantic relationships and potential annotation errors in multimodal data, resulting in inconsistent output data quality and severely impacting the training effectiveness and generalization ability of subsequent artificial intelligence models. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides an interactive intelligent annotation method and system for multimodal data, solving the problem of "low work efficiency" mentioned in the background technology.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides an interactive intelligent annotation method for multimodal data, comprising the following steps: S1, acquire multimodal data to be labeled, wherein the multimodal data to be labeled includes one or more of image data, text data, audio data, video data, 3D point cloud data, and sensor data; S2, preprocess the multimodal data to be labeled, and convert the heterogeneous original multimodal data into a unified standardized format; S3, based on the multimodal fusion model, performs preliminary intelligent annotation on the preprocessed multimodal data, and generates preliminary annotation results and corresponding confidence scores; S4, Identify low-confidence regions or inconsistent regions between modes in the preliminary annotation results; S5, Generate an interactive annotation interface to guide manual collaborative annotation and correction of the low-confidence region or the region of inconsistency between modes; S6, collect and process manual annotation and correction feedback, and merge the effective manual feedback data with the preliminary annotation results; S7, verification and optimization of annotation results based on human feedback; S8 outputs the final high-quality multimodal annotation data.

[0007] Preferably, in step S1, obtaining the multimodal data to be labeled includes: S101, Receive raw image data from a specified data source, wherein the image data is a two-dimensional image or a multispectral image, and the file format is JPEG, PNG or TIFF; S102, Receive raw text data from a specified data source, wherein the text data is natural language text or structured text, and the encoding format is UTF-8; S103, receive raw other modal data from a specified data source, wherein the other modal data is one or more of audio, video or point cloud, the audio data format is WAV or MP3, the video data format is MP4 or AVI, and the point cloud data format is PLY or LAS; S104 performs preliminary straightening and timestamp recording on the received data of different modalities to ensure that each piece of data includes a unique identifier, data type information, source information, and a timestamp accurate to the millisecond level.

[0008] Preferably, in step S2, the preprocessing of the multimodal data to be labeled includes: S201, perform timestamp calibration and sequence alignment, and use dynamic time warping algorithm or sliding window correlation analysis to unify the sampling frequency of all modes to the preset reference frequency; S202 performs intramodal data cleaning and format unification, denoises image data, adjusts brightness and contrast, segments text data, removes stop words, and converts all data into floating-point matrix or tensor format; S203 performs intermodal feature space mapping and dimension matching, mapping the original feature spaces of different modalities to a shared low-dimensional feature space through linear projection or nonlinear transformation. S204 stores the preprocessed multimodal data into a unified data structure, encodes it into a multidimensional tensor containing time, space and feature dimension information, and adds metadata.

[0009] Preferably, in step S3, the preliminary intelligent annotation based on the multimodal fusion model includes: S301, The preprocessed multimodal data is input into the multimodal feature extraction module, which includes an independent modality encoder to extract high-level semantic features from each modality of data; S302, the extracted features are deeply fused through a multimodal fusion network, wherein the fusion network adopts a cross-attention mechanism; S303, Automatic prediction for specific annotation tasks based on fused features; S304, Generate preliminary annotation results and their corresponding confidence scores, wherein the confidence scores are calculated based on the probability distribution output by the model.

[0010] Preferably, in step S4, identifying low-confidence regions or intermodal inconsistencies includes: S401, Evaluate the confidence level of each annotation item in the preliminary annotation results, and initially mark the annotation items that are lower than the preset confidence threshold; S402, detect the semantic or spatial consistency between the annotation information corresponding to different modalities; S403, mark the items with confidence levels below the preset threshold and add them to the list to be corrected; S404, Mark the areas where there are inconsistencies and conflicts between modes and add them to the list to be corrected; S405, compile a list of areas to be corrected and prioritize them by severity.

[0011] Preferably, in step S5, generating an interactive annotation interface and guiding manual collaborative annotation and correction includes: S501, dynamically construct an interactive annotation interface based on the multimodal data characteristics and annotation task type in the list of regions to be corrected; S502 highlights low-confidence labels or modal inconsistency areas in the interface with visual elements and provides prompts. S503 provides cross-modal interactive display and editing tools as well as a variety of annotation toolsets; S504 guides annotators to manually correct and supplement annotations in designated areas through interactive methods.

[0012] Preferably, in step S6, collecting and processing manual annotation and correction feedback includes: S601 can acquire all operation commands and input information of the annotator in real time on the interactive interface; S602, Verify the format and semantic validity of manually corrected data; S603 merges the validated and valid human feedback data with the preliminary annotation results, and the manually corrected data takes priority over the preliminary annotation data generated by the model. S604 records each manual correction operation and related information, generating a labeling log.

[0013] Preferably, in step S7, the verification and optimization of the annotation results based on human feedback includes: S701 uses the manually corrected annotation data as the gold standard, compares and analyzes it with the preliminary annotation results, and calculates evaluation indicators. S702, Based on the evaluation results, the parameters of the multimodal fusion model are updated or adjusted using the gradient descent method; S703, perform a final consistency and completeness check on the corrected annotation results; S704 generates an annotation quality report, recording the annotation accuracy, manual correction ratio, and overall consistency score for each modality.

[0014] Preferably, in step S8, outputting the final high-quality multimodal annotation data includes: S801 organizes the final confirmed multimodal annotation data into a standard output format; S802 transmits labeled data to the target storage system or application via a data interface; S803 generates a data list and version information, including a unique batch identifier, version number, data file path list, total data volume, and file hash checksum.

[0015] Preferred interactive intelligent annotation systems for multimodal data include: The system comprises the following modules: a data acquisition module for acquiring multimodal data to be labeled; a data preprocessing module for preprocessing the multimodal data; an intelligent labeling module for performing preliminary intelligent labeling on the preprocessed multimodal data based on a multimodal fusion model; an anomaly detection module for identifying low-confidence regions or inconsistencies between modalities in the preliminary labeling results; an interactive labeling interface generation module for generating an interactive labeling interface; a human feedback processing module for collecting and processing human annotation and correction feedback; a labeling verification and optimization module for verifying and optimizing the labeling results based on human feedback; and a data output module for outputting the final high-quality multimodal labeled data. These modules are interconnected and work collaboratively.

[0016] (III) Beneficial Effects This invention provides an interactive intelligent annotation method and system for multimodal data. It has the following beneficial effects: (1) This invention performs preliminary intelligent annotation through a multimodal fusion model and guides manual correction of low confidence and inconsistent regions between modalities through an interactive interface. This reduces the overall workload of manual annotation, avoids frequent switching between different modal operation interfaces, solves the problems of low efficiency, high cost and long cycle of multimodal data annotation in the prior art, and improves annotation efficiency.

[0017] (2) The present invention identifies low confidence regions and inconsistencies between modalities through an anomaly detection module, and verifies the validity of the corrected data through a manual feedback processing module. At the same time, the model is optimized, forming a closed-loop verification and correction mechanism from automatic detection to manual correction and model iteration. This effectively reduces the labeling deviation caused by the complexity of multimodal semantic understanding, improves the quality and stability of the labeled data, and overcomes the defects of the prior art, such as the lack of an effective verification and correction mechanism and the inconsistent quality of output data.

[0018] (2) This invention achieves time synchronization, spatial alignment and format unification of multimodal data through data preprocessing, captures deep semantic associations between modalities with the help of multimodal fusion models, and the interactive interface supports cross-modal linkage editing, which enhances the collaborative annotation capability between different modal data, ensures that the annotation results can accurately reflect cross-modal semantic associations, provides high-quality training data for subsequent artificial intelligence models, and improves the training effect and generalization ability of the models. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2This is a schematic diagram of the system framework of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 - Figure 2 This invention provides an interactive intelligent annotation method for multimodal data, comprising the following steps: S1. Acquire multimodal data to be labeled. Collect raw multimodal data from multiple data sources. Multimodal data to be labeled refers to the raw data set that has not yet been labeled by humans or machines, including one or more of the following forms: image data, text data, audio data, video data, 3D point cloud data, sensor data, etc.

[0022] The concrete paths of data sources include, but are not limited to: image data can come from vehicle cameras, monitoring equipment, medical imaging equipment, or satellite remote sensing systems; text data can come from web crawling, user comment databases, news corpora, or literature databases; audio data can come from speech recognition systems, environmental monitoring devices, or smart assistant interaction records; video data can come from autonomous driving recorders, live streaming media, or security monitoring systems; 3D point cloud data can come from LiDAR scanners, depth cameras, or the output of 3D reconstruction algorithms; sensor data can come from IoT devices, industrial control systems, or smart wearable devices. During the acquisition process, preliminary unified management of data from different sources is required. To ensure the integrity and reliability of data acquisition, the system should achieve a stable connection with the data source and have a data transmission verification mechanism, such as using cyclic redundancy check or hash check, to verify that the data has not been damaged or lost during transmission. Specifically, this includes the following steps: S101, Receive raw image data from a specified data source. The image data can be a two-dimensional image or a multispectral image, and its file format can be JPEG, PNG or TIFF. S102, Receive raw text data from a specified data source. The text data can be natural language text or structured text, and the encoding format can be UTF-8. S103, Receive raw other modal data from the specified data source. Other modal data can be one or more of audio, video or point cloud. The audio data format can be WAV or MP3, the video data format can be MP4 or AVI, and the point cloud data format can be PLY or LAS. S104 performs preliminary straightening and timestamp recording on the received data of different modalities to ensure that each piece of data includes a unique identifier, data type information, source information, and a timestamp accurate to the millisecond level.

[0023] S2 preprocesses the multimodal data to be labeled, primarily converting the heterogeneous raw multimodal data into a unified, standardized format suitable for subsequent intelligent annotation models. Preprocessing is a crucial step in ensuring the quality and consistency of multimodal data. Specific operations include data cleaning, format conversion, time synchronization, spatial alignment, missing value handling, and outlier detection. In the data cleaning stage, noise, redundant information, or corrupted data segments are removed. Examples include blurred frames or overexposed areas in image data, garbled characters or stop words in text data, and background noise in audio data. Format conversion unifies raw data from different sources and in different formats into a standard format acceptable to the model. For example, all images are uniformly adjusted to a specific resolution and color space, and all text is uniformly encoded. Time synchronization is a challenging aspect of multimodal data preprocessing because different sensors or data streams may record data at different sampling frequencies or start times. Dynamic time warping algorithms or frequency domain alignment techniques based on Fourier transform can be used to accurately match the time axes of all modal data, ensuring that data collected at the same time point can be correctly correlated. Spatial alignment targets modalities with spatial attributes, such as image and point cloud data. Through camera intrinsic and extrinsic parameter calibration or feature point matching algorithms, data from different sensor coordinate systems are transformed to a unified spatial coordinate system. Missing value handling can employ interpolation, mean imputation, or deep learning completion models. Outlier detection can identify and process data points deviating from the normal range based on statistical methods or machine learning models, preventing them from negatively impacting annotation accuracy. Data preprocessing parameters come from pre-defined industry standards, empirical values ​​specific to application scenarios, or dataset characteristics derived through statistical analysis. Specifically, the following steps are included: S101, perform timestamp calibration and sequence alignment, using dynamic time warping algorithm or sliding window correlation analysis to ensure that the timestamps of each modality data correspond accurately, and unify the sampling frequency of all modalities to the preset reference frequency; S102, perform intramodal data cleaning and format unification, such as denoising and adjusting brightness and contrast of image data, segmenting and removing stop words of text data, and converting all data into floating-point matrix or tensor format; S103, perform intermodal feature space mapping and dimension matching, for example, by mapping the original feature spaces of different modalities to a shared low-dimensional feature space through linear projection or nonlinear transformation, in order to eliminate dimensional differences and promote subsequent fusion; S104 stores the preprocessed multimodal data into a unified data structure, such as encoding image, text, and audio data into a multidimensional tensor, which includes time, space, and feature dimension information, and attaches metadata, such as the original data identifier and preprocessing parameters.

[0024] S3, based on a multimodal fusion model, performs initial intelligent annotation. This involves using an artificial intelligence model to automatically or semi-automatically annotate pre-processed multimodal data, significantly improving annotation efficiency and providing a foundation for subsequent manual refinement. The multimodal fusion model is the core component of this method and can employ various architectures, such as early fusion, late fusion, or hybrid fusion. Early fusion involves concatenating the raw data or low-level features from different modalities during the feature extraction stage and then inputting them into a unified network for processing. Late fusion processes each modal data independently and then fuses them at the output or decision layer. Hybrid fusion combines the advantages of both approaches. The model here can be a deep learning model, such as a multimodal encoder-decoder model based on the Transformer architecture, or a model combining convolutional neural networks and recurrent neural networks. During training, the model learns how to extract meaningful features from different modal data and understand their semantic relationships, thereby enabling accurate annotation predictions. Annotation tasks can include object detection, image segmentation, text entity recognition, sentiment analysis, event recognition, and cross-modal semantic association. The initial annotation results include the category, location, and attribute information of the labeled objects, as well as a key confidence score, which quantifies the model's certainty about its own predictions. The confidence score can be calculated based on the probability distribution of the model's output layer, for example, by using a softmax function to obtain the probability values ​​for each category and taking the maximum probability value as the confidence score. For example, for an image object detection task, the model might output bounding box coordinates, object category, and the probability value for that category. Parameter sources include the weights of the pre-trained model, the type of loss function used during model training, and optimizer parameters. The validation mechanism includes statistical analysis of the probability distribution of the model output to ensure that the confidence score is interpretable and discriminative. Specifically, this includes the following steps: S301, the preprocessed multimodal data is input into the multimodal feature extraction module, which includes independent modal encoders (e.g., ResNet for images and BERT for text) to extract high-level semantic features from each modal data respectively; S302 uses a multimodal fusion network to deeply fuse the extracted features. The fusion network can employ a cross-attention mechanism to enable the feature vectors of different modalities to interact and capture intermodal correlation information. The fusion process can be represented by the following formula: in, , as well as This represents the feature vector extracted from modalities such as images, text, and audio. This represents the fused multimodal feature vector.

[0025] S303, Automatic prediction for specific annotation tasks based on fused features, such as object detection in images, entity recognition in text, or behavior classification in videos; S1034, Generating preliminary annotation results and their corresponding confidence scores. The confidence scores can be calculated based on the probability distribution output by the model. For example, for a classification task, the confidence score C can be expressed as: in, It is the probability that the model predicts that a sample belongs to class k.

[0026] S4 identifies low-confidence regions or intermodal inconsistencies in the initial annotation results. Automated methods are used to filter out regions requiring manual intervention, improving efficiency and accuracy. Low-confidence regions refer to predictions output by the model during initial annotation where the confidence score is below a preset threshold. Intermodal inconsistencies refer to regions where different modalities have conflicting or contradictory annotation information for the same semantic or entity. Various algorithms can be used for identification. The confidence threshold setting needs to comprehensively consider the difficulty of the annotation task, data quality, and accuracy requirements. The value range is between 60% and 95%. Intermodal inconsistency detection can utilize cross-modal semantic matching algorithms or logical reasoning rules. For example, a cross-modal knowledge graph can be constructed to detect conflicts between objects in images and entities in text descriptions. Spatial conflict detection can be used for data with geographical or spatial attributes, such as differences in the spatial location or size of the same object in images and point cloud data. Conflict types can be subdivided into semantic conflicts, spatial conflicts, and quantitative conflicts. The validation mechanism includes sensitivity analysis of the threshold setting and evaluation of the recall and precision of the inconsistency detection algorithm. Specifically, it includes the following steps: S401, Evaluate the confidence level of each annotation item in the preliminary annotation results, for example, compare the confidence level of all annotation items with a preset confidence level threshold (e.g., 70%), and the annotation items below this threshold are initially marked. S402, detect the semantic or spatial consistency between the annotation information corresponding to different modalities, for example, by calculating the semantic similarity score between the image target detection result and the corresponding entity in the text description, or by comparing the geometric size and position difference of the same target in the image and the point cloud. S403, mark the annotation items with confidence scores below the preset threshold, add them to the list to be corrected, and record their specific location, annotation content and confidence score value; S404, Mark the annotation areas where there are inconsistent conflicts between modalities. For example, when the semantic similarity between the image detection result and the text description is lower than a specific threshold, or when there are significant geometric deviations of the same target in different modalities after spatial alignment, add it and all its associated modality annotation information to the list of areas to be corrected; S405, Summarize the list of areas to be corrected that require manual intervention, and prioritize them according to the low confidence level or the severity of modality inconsistency.

[0027] S5, Generate an interactive annotation interface and guide manual collaborative annotation and correction, mainly providing an intuitive, efficient, and multi-modal collaborative operation working environment for annotators. The interactive annotation interface has a high degree of visualization and operability, and the layout of the interface should clearly present all relevant modality data. For example, images, texts, audio waveforms, video frame sequences, and 3D point cloud views can be synchronously displayed in different sub-windows. For the low-confidence annotations or modality-inconsistent areas identified in S4, the system will prominently highlight them in the interface, directly guiding the attention of the annotators. The interface should provide rich interactive elements and collaborative tools. The key lies in realizing cross-modal linkage display and editing: when the annotator operates on a certain modality, the corresponding areas on other associated modalities should also be synchronously displayed or updated. The real-time preview function allows the annotator to instantly view the modification effect during the correction process, improving the accuracy of the correction. The operation log recording mechanism ensures that every manual correction operation, its time, the identity of the annotator, and the content before and after the modification are completely recorded, providing data for subsequent quality traceability and model optimization. The parameter sources include the interface configuration template, user preference settings, and preset annotation specifications. The verification mechanism includes the interface element response speed test, multi-modal linkage synchronization test, and user experience evaluation. Specifically, it includes the following steps: S501, Dynamically construct an interactive annotation interface according to the multi-modal data characteristics and annotation task types in the list of areas to be corrected; S502, Highlight the low-confidence annotations or modality-inconsistent areas marked in S4 with visual elements in the interface, and at the same time provide concise hint information to explain their possible problems; S503, Provide cross-modal linkage display and editing tools. For example, when adjusting the target bounding box on an image, the corresponding entity mention in the text description window will be synchronously located and allowed to be edited, and vice versa; provide a variety of annotation tool sets, such as pixel-level segmentation tools for images and entity relationship graph construction tools for texts; S504, Guide the annotator to manually correct and supplement annotations for the specified area through intuitive interaction methods. For example, the annotator can modify the bounding box, change the text label, or add new semantic associations through operations such as dragging, clicking, and inputting.

[0028] S6 collects and processes feedback from manual annotations and corrections. Its primary responsibility is to capture and organize all operations performed by annotators on the interactive annotation interface, transforming them into structured feedback data and performing preliminary validity verification. Feedback data includes, but is not limited to: newly added annotation objects, modified annotation attributes, deleted erroneous annotations, resolutions of intermodal consistency conflicts, and supplementary annotations added by annotators. The system needs to acquire these operation instructions and input information in real time. For example, when an annotator draws a new bounding box on an image, the system immediately records the bounding box's coordinates, category label, and creation time; when an annotator modifies a word in the text description, the system records the text content before and after the modification. To ensure the validity of the feedback data, the system performs format and semantic verification. Format validity verification includes checking whether the input coordinates conform to the image range, whether the text is empty, and whether the category label is in a predefined list. Semantic validity verification can utilize a lightweight semantic model to preliminarily determine whether the manually corrected text description significantly conflicts with the image content. Validated manual feedback data is then merged with the preliminary annotation results, replacing or updating the corresponding annotation items. To ensure traceability, every manual correction operation, along with its timestamp, the person responsible for annotation, and the content before and after the modification, is meticulously recorded in the operation log. The log is used not only for quality control but also provides valuable data for subsequent model retraining and problem analysis. Parameter sources include predefined annotation format specifications, data type validation rules, and user permission configurations. Validation mechanisms include real-time data structure validation and conflict detection and resolution for multi-user concurrent operations, such as using optimistic locking or distributed transactions to ensure data consistency. Specifically, the following steps are included: S601 can acquire all operation instructions and input information of the annotator on the interactive interface in real time, including but not limited to adding, deleting and modifying bounding boxes, changing category labels, editing text content, and establishing or removing cross-modal associations. S602, verify the format and semantic validity of manually corrected data. For example, check whether the coordinates of the newly added bounding box are valid, whether the modified text contains illegal characters, and perform a preliminary match with the preset dictionary or rules to eliminate obvious input errors. S603 merges the validated and valid human feedback data with the preliminary annotation results using transactional operations to ensure the atomicity and consistency of data updates. For the same annotation item, the manually corrected data will take priority over the preliminary annotation data generated by the model. S604 records each manual correction operation, its timestamp, and the identity of the annotator, generating a detailed annotation log. This log can be used for subsequent auditing, tracing, and model performance analysis.

[0029] S107, based on human feedback, verifies and optimizes annotation results. It primarily utilizes human expertise to deeply validate the overall annotation results, using this feedback to improve the intelligent annotation model and continuously enhance annotation quality. The manually corrected annotation data serves as the "gold standard" or "true value," compared with the initial annotation results. Evaluation metrics such as precision, recall, F1 score, and intersection-over-union ratio are calculated to quantify the intelligent annotation model's performance across different tasks and modalities. For low-confidence or modal inconsistencies, manually corrected data can directly correct model errors. This corrected data can be integrated into the training set for incremental learning or fine-tuning of the multimodal fusion model. Through iterative optimization, the model gradually learns the implicit knowledge and correction patterns of human annotators, reducing the probability of errors in similar scenarios. During model optimization, federated learning or active learning strategies can be employed, using only a small amount of high-quality manually corrected data for efficient fine-tuning, avoiding overfitting. The validation process also includes a final consistency and completeness check on the corrected annotation results. Finally, a detailed annotation quality report is generated, including metrics such as accuracy, recall, manual correction rate, and consistency score for each modality, providing users and model developers with a comprehensive quality overview. Parameter sources include model training configuration parameters, evaluation metric thresholds, and validation rule sets. Validation mechanisms include model performance evaluation reports, manual review and sampling mechanisms, and data quality quantification metrics. Specifically, the following steps are included: S701 uses the manually corrected labeled data as the gold standard and compares it with the preliminary labeled results generated by S3 to calculate multiple evaluation indicators, including accuracy, recall, F1 score and intersection-over-union ratio. S702. Based on the evaluation results, the parameters of the multimodal fusion model can be updated or adjusted using the gradient descent method, with the gold standard data as a supervision signal, to fine-tune the model's weights, enabling it to predict more accurately in subsequent annotation tasks. The model parameter update can be expressed as the following formula: in, Represents model parameters, It's the learning rate. It is a loss function. It is the gold standard based on manual labeling. It is the result of model prediction. It is the input data; S703, perform a final consistency and completeness check on the corrected annotation results; S704 generates an annotation quality report, which records in detail the annotation accuracy rate, manual correction ratio, and overall consistency score for each modality.

[0030] S8 outputs the final high-quality multimodal labeled data. This data, optimized through intelligent assistance, manual correction, and rigorous verification, is delivered to downstream applications or storage systems in a standardized format. The final output multimodal labeled data is a high-quality training dataset for artificial intelligence models, characterized by high accuracy, high consistency, and high completeness. Specifically, it includes the following steps: S801 organizes the final confirmed high-quality multimodal annotation data into a standard output format, for example, storing all image annotation information in JSON format and text annotation information in XML format, and ensuring that there are clear association references between them; S802 transmits labeled data to the target storage system or application via a data interface. For example, it packages the data and uploads it to a specified data lake or model training platform via a file transfer protocol or API interface. S803 generates a data list and version information, including a unique batch identifier for the output data, the current version number, a list of all data file paths, the total amount of data, and the hash checksum of all files, to facilitate data management and verification.

[0031] Based on the above, this invention also provides an interactive intelligent annotation system for multimodal data, including a data acquisition module, a data preprocessing module, an intelligent annotation module, an anomaly detection module, an interactive annotation interface generation module, a manual feedback processing module, an annotation verification and optimization module, and a data output module. These modules are interconnected and work collaboratively. Specifically: The data acquisition module is used to acquire multimodal data to be labeled. It is responsible for establishing connections with external data sources and continuously receiving raw multimodal data according to a pre-defined strategy. It includes several sub-modules for different data types, such as image data receivers, text data receivers, audio data receivers, and point cloud data receivers. The data acquisition module supports various data transmission protocols, such as HTTP, FTP, Kafka, or message queues. Received data is first cached and undergoes preliminary integrity checks, such as verification based on file size, header information, or metadata, to ensure that the data has not been corrupted or lost during transmission.

[0032] The data preprocessing module is used to preprocess multimodal data to be labeled. It receives the raw data output from the data acquisition module and performs a series of transformation operations, including a data cleaning unit, a format conversion unit, a time synchronization unit, and a spatial alignment unit. The data cleaning unit identifies and removes noise or outliers from the data. The format conversion unit unifies various heterogeneous data formats into a standard format. The time synchronization unit uses advanced algorithms, such as dynamic time warping or cross-correlation analysis, to ensure accurate time alignment between different modalities. The spatial alignment unit unifies data with spatial attributes into a shared coordinate system. The preprocessed data is then organized into a unified multidimensional tensor or structured data for use by subsequent modules.

[0033] The intelligent annotation module performs preliminary intelligent annotation based on a multimodal fusion model. This includes a pre-trained multimodal fusion model, which is a deep neural network, such as a Transformer model or a combination of graph neural networks and convolutional neural networks. The intelligent annotation module receives standardized multimodal data output from the data preprocessing module, extracts high-level semantic features from each modality using a multimodal feature extractor, then deeply fuses these features using a multimodal fusion network, and finally automatically generates preliminary annotation results and their corresponding confidence scores using an annotation predictor. This module supports various annotation task types, such as object detection, semantic segmentation, entity recognition, and sentiment analysis.

[0034] The anomaly detection module identifies low-confidence regions or inconsistencies between modalities in the initial annotation results. It receives the initial annotation results and their confidence scores from the intelligent annotation module. The module includes a confidence assessment unit and a modal consistency verification unit. The confidence assessment unit compares the confidence of each annotation item with a preset threshold, marking low-confidence regions below the threshold. The modal consistency verification unit uses cross-modal semantic matching algorithms or logical rules to detect semantic conflicts or spatial contradictions in the annotation information between different modalities. All identified anomaly regions are recorded in detail and sorted according to a preset priority, forming a list awaiting manual correction.

[0035] The interactive annotation interface generation module is used to generate interactive annotation interfaces. Based on the list of interfaces requiring manual correction output by the anomaly detection module, this module dynamically constructs a highly visual interactive annotation interface. The module is responsible for rendering the interface layout, synchronously displaying data from various modalities, and highlighting areas requiring manual intervention in a prominent manner. It includes a variety of multimodal annotation toolsets, such as image bounding box drawing tools, text entity selection tools, audio waveform editing tools, and 3D point cloud manipulation tools. The interactive annotation interface generation module also implements cross-modal linkage functionality, ensuring that actions performed by an annotator in one modality are reflected in other related modalities in real time, improving the efficiency and accuracy of collaborative annotation.

[0036] The manual feedback processing module collects and processes manual annotation and correction feedback. This module is responsible for real-time monitoring and capturing all operations and inputs from annotators on the interactive annotation interface. The module includes a feedback receiving unit and a data validation unit. The feedback receiving unit transforms the annotators' modifications, additions, or deletions into structured feedback data. The data validation unit verifies the format and semantic validity of the received feedback data to ensure its accuracy and standardization. Validated feedback data is integrated into the current annotation results, generating a detailed operation log that records the details, timestamp, and annotator's identity for each modification.

[0037] The annotation verification and optimization module is used to verify and optimize annotation results based on human feedback. This module receives manually corrected data from the human feedback processing module, including a quality assessment unit and a model optimization unit. The quality assessment unit uses the manually corrected data as the gold standard to perform a comprehensive quality assessment of the initial annotation results from the intelligent annotation module, calculating various performance indicators. The model optimization unit uses the manually corrected data to incrementally learn or fine-tune the multimodal fusion model in the intelligent annotation module. Through iterative training, it updates the model parameters, allowing it to gradually learn the human correction patterns and continuously improve the model's intelligent annotation capabilities.

[0038] The data output module is used to output the final high-quality multimodal annotation data. It is responsible for formatting and delivering the final multimodal annotation data, which has undergone intelligent assistance, manual correction, and rigorous verification and optimization. The data output module supports multiple standard output formats, such as COCO, PascalVOC, JSONLines, or XML. Specifically, it includes a format conversion unit and a data interface unit. The format conversion unit converts the internal data structure into the user-specified output format. The data interface unit transmits the annotation data to the target storage system or downstream applications through standard data transmission protocols or APIs, while generating detailed data lists and version information to ensure data traceability and usability.

[0039] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An interactive intelligent annotation method of multi-modal data, characterized in that, The method comprises the following steps: S1, obtaining multi-modal data to be labeled, the multi-modal data to be labeled comprising one or more of image data, text data, audio data, video data, three-dimensional point cloud data, and sensor data; S2, preprocessing the multi-modal data to be labeled to convert heterogeneous original multi-modal data into a unified standardized format; S3, performing preliminary intelligent labeling on the preprocessed multi-modal data based on a multi-modal fusion model to generate a preliminary labeling result and a corresponding confidence score; S4, identifying low-confidence regions or inter-modal inconsistent regions in the preliminary labeling result; S5, generating an interactive labeling interface to guide manual collaborative labeling and correction of the low-confidence regions or inter-modal inconsistent regions; S6, collecting and processing manual labeling and correction feedback, and merging effective manual feedback data with the preliminary labeling result; S7, verifying and optimizing the labeling result based on the manual feedback; S8, outputting final high-quality multi-modal labeling data. 2.The method for interactive intelligent annotation of multi-modal data according to claim 1, characterized in that: In S1, obtaining multi-modal data to be labeled comprises: S101, receiving original image data from a specified data source, the image data being two-dimensional images or multispectral images, and the file format being JPEG, PNG, or TIFF; S102, receiving original text data from a specified data source, the text data being natural language text or structured text, and the encoding format being UTF-8; S103, receiving original other modal data from a specified data source, the other modal data being one or more of audio, video, or point cloud, the audio data format being WAV or MP3, the video data format being MP4 or AVI, and the point cloud data format being PLY or LAS; S104, performing preliminary regularization and timestamp recording on the received different modal data to ensure that each piece of data includes a unique identifier, data type information, source information, and a timestamp accurate to the millisecond level. 3.The method of claim 1, wherein: In S2, preprocessing the multi-modal data to be labeled comprises: S201, performing timestamp calibration and sequence alignment, and using a dynamic time warping algorithm or a sliding window correlation analysis to unify the sampling frequencies of all modalities to a preset reference frequency; S202, performing intra-modal data cleaning and format unification, denoising image data, adjusting brightness and contrast, tokenizing text data, removing stop words, and converting all data into a floating-point matrix or tensor format; S203, performing inter-modal feature space mapping and dimension matching, and mapping the original feature spaces of different modalities to a shared low-dimensional feature space through linear projection or nonlinear transformation; S204, storing the preprocessed multi-modal data as a unified data structure, encoding it as a multi-dimensional tensor containing time, space, and feature dimension information, and attaching metadata. 4.The method of claim 1, wherein: In S3, performing preliminary intelligent labeling based on a multi-modal fusion model comprises: S301, inputting the preprocessed multi-modal data into a multi-modal feature extraction module, the module comprising independent modal encoders that extract high-level semantic features from each modality of data; S302, deep fusion of the extracted features through a multi-modal fusion network, the fusion network adopting a cross-attention mechanism; S303, automatic prediction of a specific annotation task based on the fused features; S304, generation of a preliminary annotation result and a corresponding confidence score, the confidence score being calculated according to a probability distribution output by the model. 5.The method for interactive smart annotation of multi-modal data of claim 1, wherein: In S4, identifying low-confidence regions or inter-modal inconsistent regions includes: S401, evaluating the confidence of each annotation item in the preliminary annotation result, and preliminarily marking annotation items with a confidence lower than a pre-set confidence threshold; S402, detecting semantic or spatial consistency between annotation information corresponding to different modalities; S403, marking annotation items with a confidence lower than a pre-set threshold and adding them to a list of regions to be corrected; S404, marking annotation regions with inter-modal inconsistent conflicts and adding them to the list of regions to be corrected; S405, summarizing the list of regions to be corrected and prioritizing them according to severity.

6. The method of interactive smart labeling of multi-modal data as claimed in claim 1 wherein: In S5, generating an interactive annotation interface and guiding human collaborative annotation and correction includes: S501, dynamically constructing an interactive annotation interface according to the multi-modal data characteristics and annotation task types in the list of regions to be corrected; S502, highlighting low-confidence annotations or inter-modal inconsistent regions in the interface with visual elements and providing prompt information; S503, providing cross-modal linkage display and editing tools and a variety of annotation tools; S504, guiding annotation personnel to manually correct and supplement annotations for specified regions through interactive methods.

7. The method of interactive intelligent annotation of multi-modal data as claimed in claim 1, wherein: In S6, collecting and processing human annotation and correction feedback includes: S601, real-time acquisition of all operation instructions and input information of the annotation personnel on the interactive interface; S602, verification of the format and semantic validity of the human correction data; S603, merging of the valid human feedback data that passes the verification with the preliminary annotation result, with human correction data taking priority over the preliminary annotation data generated by the model; S604, recording of each human correction operation and related information to generate an annotation log. 8.The method for interactive smart labeling of multi-modal data of claim 1, wherein: In S7, verification and optimization of the annotation result based on human feedback includes: S701, comparison and analysis of the human-corrected annotation data as a gold standard with the preliminary annotation result to calculate evaluation indicators; S702, updating or adjusting the parameters of the multi-modal fusion model using the gradient descent method according to the evaluation results; S703, final consistency and completeness verification of the corrected annotation result; S704, generation of an annotation quality report, recording the annotation accuracy, human correction proportion, and overall consistency score of each modality. 9.The method of claim 1, wherein: In S8, output of final high-quality multi-modal annotation data includes: S801, arrangement of the final confirmed multi-modal annotation data into a standard output format; S802, transmission of the annotation data to a target storage system or application through a data interface; S803, generation of a data list and version information, including a unique batch identifier, version number, data file path list, data total amount, and file hash checksum.

10. The interactive intelligent labeling system for multi-modal data of any one of claims 1-9, wherein: It includes: a data acquisition module for acquiring multi-modal data to be annotated; a data preprocessing module for preprocessing the multi-modal data to be annotated; The intelligent labeling module is configured to perform preliminary intelligent labeling on the preprocessed multi-modal data based on a multi-modal fusion model. Abnormalities The interactive labeling interface generation module is configured to generate an interactive labeling interface. The artificial feedback processing module is configured to collect and process artificial labeling and correction feedback. The labeling verification and optimization module is configured to perform verification and optimization of the labeling result based on the artificial feedback. The data output module is configured to output final high-quality multi-modal labeling data. The data acquisition module, the data preprocessing module, the intelligent labeling module, the abnormality detection module, the interactive labeling interface generation module, the artificial feedback processing module, the labeling verification and optimization module, and the data output module are connected through communication and work cooperatively.

Images