A multi-party cross-domain collaborative data labeling method and system and a storage medium

Abstract

This invention relates to the field of data annotation technology, and in particular to a multi-party cross-domain collaborative data annotation method, system, and storage medium. The method sets up a modality adaptation correction unit, configures several annotation processing unit interfaces corresponding to different modality types, extracts the output tensors of each model and maps them to a unified data structure, label system, and coordinate representation to form a standard annotation format. This unifies the heterogeneous pre-annotation results output by different pre-trained models into a standardized format, enabling subsequent confidence assessment, task allocation, and result verification modules to process multimodal data uniformly. This avoids the additional conversion overhead caused by format incompatibility and effectively improves the system processing efficiency and accuracy in multimodal annotation scenarios.

Description

Technical Field

[0001] This invention relates to the field of data annotation technology, and in particular to a multi-party cross-domain collaborative data annotation method, system, and storage medium. Background Technology

[0002] With the large-scale application of artificial intelligence technology, data annotation, as the core support for AI model training, has seen its demand rapidly evolve from single-modality, single-subject to multi-modality (image, text, voice, etc.) and cross-domain collaboration among multiple business entities.

[0003] Currently, there are two main methods for implementing data annotation technology, both of which have significant technical shortcomings: Firstly, centralized annotation schemes require all raw data from various business entities to be centralized on a single annotation platform. This not only poses risks of data privacy leaks and difficulty in defining data sovereignty, but also makes it difficult to guarantee the compliance and security of data transmission when the network environments of different business entities are cross-regional. In addition, centralized architectures are prone to load bottlenecks, resulting in low annotation efficiency.

[0004] Secondly, while distributed annotation schemes enable local processing of raw data and avoid the risk of data leakage associated with centralized annotation, they lack a unified and trustworthy collaborative mechanism. When working across different business entities' network environments (cross-domain), issues arise such as unreasonable task allocation, poor consistency of annotation results, lack of data security control, and difficulty in guaranteeing annotation quality.

[0005] Currently, there is no existing technology that deeply integrates trusted data space technology with multi-party cross-domain collaborative data annotation. There is only a technical approach that uses trusted data space as a basic carrier and simply overlays conventional annotation processes. Summary of the Invention

[0006] To address this, the present invention provides a multi-party cross-domain collaborative data annotation method, which aims to overcome to some extent the shortcomings of existing multi-party cross-domain data annotation technologies, such as privacy leakage, inefficient collaboration, insufficient security and compliance, low annotation quality and efficiency, as well as the shortcomings of existing technical solutions that simply splice together trusted data spaces and conventional annotation processes, lack of substantial innovation, and inability of pre-annotation models to continuously self-optimize. This invention aims to solve the problem of reduced annotation quality caused by at least one of the above-mentioned issues during the data annotation process.

[0007] To achieve the above objectives, on the one hand, this invention provides a multi-party cross-domain collaborative data annotation method, which is applied to the implementation of a trusted data space. Based on the trusted data space implementation, it reuses the core modules of the trusted data space, such as identity authentication, access control, encrypted storage, secure transmission, auditing and clearing, and privacy computing, including: Step S1: Input the pre-trained model as the main body; Step S2, mark the task initialization configuration; Step S3: Local de-identification processing of the dataset to be labeled; Step S4: Local pre-annotation and cross-domain annotation task allocation; Step S5: Human-machine collaboration completes data annotation through a sandbox. Step S6: Synchronize and verify the consistency of annotation results; Step S7: The algorithm of the AI ​​pre-labeled model is updated iteratively. Step S8: Task acceptance and destruction of labeled data; The steps for local pre-labeling for a single period using a data platform include: Based on the type and scenario of the data to be labeled, determine the corresponding labeling processing units; The annotation processing unit is used to pre-annotate the data to be annotated locally, and the pre-annotation results of different modalities are unified into a standard format; The confidence level of each data to be labeled is determined based on the pre-labeling results, and the pre-labeling results are divided into accepted data and re-verification data based on the confidence level threshold. The annotation processing unit sends the re-verification data to the data source and performs re-verification annotation; Determine the results of the retest annotations and determine the retest confidence level for each retest data; The annotation processing unit is iterated to adapt to the verification confidence level; Each annotation processing unit is equipped with the corresponding pre-trained model and has several corresponding interfaces. Using the aforementioned confidence level, data exceeding a preset quality threshold are selected as high-quality labeled data and set as the incremental training dataset. The original data is retained locally by the data provider, and only the updated model parameter values ​​encrypted with homomorphic encryption technology are transmitted to perform incremental training on the pre-trained model carried by the annotation processing unit to complete the algorithm's self-iterative update.

[0008] Furthermore, the data platform is equipped with a modality adaptation correction unit to unify the pre-annotation results of different modalities into a standard format; The modal adaptation and correction unit is configured with several annotation processing unit interfaces corresponding to different modal types. This includes an image interface for image data, a text interface for text data, and a voice interface for voice data. When the corresponding annotation processing unit is invoked to perform pre-annotation, the modality adaptation correction unit extracts the output tensors of each model and maps them to the standard annotation format; The standard annotation format includes a unified data structure, label system, and coordinate representation method, which are used to unify the format of the pre-annotation results.

[0009] Furthermore, when determining the confidence level of each piece of data to be labeled, the data platform compares the pre-labeling results with the confidence scores output by a preset reference standard or the model itself, including: Calculate the pre-labeling accuracy of each labeled object, and generate a confidence score based on the accuracy. Based on the preset first and second confidence thresholds, the pre-labeled results are divided into three levels: Results exceeding the first confidence threshold are considered high confidence levels and are directly output as the accepted data. Results that are not higher than the first confidence threshold and are higher than the second confidence threshold are considered to have a medium confidence level. Results not exceeding the second confidence threshold are classified as low confidence level; The results of both the medium confidence level and the low confidence level are included in the retest labeling. The first confidence threshold is higher than the second confidence threshold, and the standard annotation format is directly used as the input data format for confidence assessment.

[0010] Optionally, for a single annotation processing unit, methods for determining its confidence level include: Determine the accuracy of the annotation in the standard annotation format; The labeled accuracy rates are broken down and mapped to the data categories corresponding to each interface to form several interface accuracy rates corresponding to each interface. The interface adaptability of the annotation processing unit is determined based on the interface accuracy and the number of annotations completed per unit time. For a single interface, if the interface adaptability is not higher than the second confidence threshold, it is determined that the interface is not applicable to the data corresponding to the interface of that category.

[0011] Optionally, when performing a re-verification annotation task, for the confidence level of a single pre-annotation result, tasks with a confidence level lower than the second confidence threshold are marked as high-priority tasks; Tasks that fall under the high confidence level and require review are marked as ordinary priority tasks; The data platform calculates the real-time availability of each annotation processing unit based on the interface compatibility of the annotation processing unit and the corresponding number of allocated verification annotation tasks. The real-time availability is positively correlated with the interface adaptability and negatively correlated with the current task load.

[0012] Furthermore, during the iteration of the annotation processing unit, high-priority tasks are assigned to the annotation processing unit with the highest real-time availability, and... Prioritize assigning tasks of the same type to the annotation processing unit with the highest type suitability score; Monitor the continuous working time of each annotation processing unit. When the continuous working time of the corresponding device for a certain annotation processing unit exceeds the preset continuous working time threshold, The interface adaptability of the response annotation processing unit is automatically reduced, and its corresponding real-time availability is calculated to avoid performance fluctuations or processing quality degradation caused by long-term continuous operation of the device.

[0013] If the response exceeds the continuous working duration threshold, the weight value of the interface adaptability in the real-time availability calculation model or the value of the interface adaptability itself will be automatically reduced. And calculate the corresponding real-time availability based on the reduced weight value or interface adaptability value, in order to avoid performance fluctuations or processing quality degradation caused by long-term continuous operation of the equipment. When the adaptability of any interface fails to reach the second confidence threshold, the second confidence threshold is lowered. When the adaptability of any interface can reach the first confidence threshold, the first confidence threshold is increased.

[0014] On the other hand, the present invention also provides a multi-party cross-domain collaborative data annotation system, comprising: The model scheduling unit is used to retrieve the corresponding annotation processing unit according to the data annotation type and scenario, and includes an adaptation and correction unit that unifies the pre-annotation results of different modalities into a standard format. The pre-labeling evaluation unit is used to calculate the confidence level of the pre-labeling results and classify the processing level. The capability assessment unit is used to collect historical annotation data from the annotation processing unit and generate interface adaptability using a multi-index comprehensive evaluation method. The dynamic allocation unit is used to split the dataset to be labeled and dynamically allocate it based on interface adaptability, load, task attributes and pre-labeling result confidence, using a task allocation strategy with fatigue suppression mechanism.

[0015] Furthermore, the specific operation mode of the pre-annotation evaluation unit includes: Obtain the pre-annotated results after processing in a unified format, evaluate the quality of the pre-annotated results, and calculate the accuracy index that reflects the consistency between the pre-annotated results and the reference basis. Based on the accuracy index, a confidence index is generated to characterize the reliability of the pre-labeled results; The confidence index is compared with at least one preset confidence threshold. Based on the comparison result, the pre-labeled result is divided into at least two different confidence levels, and the corresponding level identifier is output.

[0016] Furthermore, when completing the confidence level classification, the pre-labeled evaluation unit performs verification based on the level identifier, including: If the response belongs to the first confidence level of the pre-labeled result, it will be output as a candidate labeling result and will not be sent to the review stage; For pre-labeled results belonging to the second confidence level, they are sent to the verification stage, and can be assigned corresponding task priorities according to their confidence level, which can be used to guide the selection of labeling processing units when allocating tasks in the future. The capability assessment unit records the confidence index and level identifier of each pre-labeled result for use during model iteration or quality control.

[0017] In addition, the present invention provides a storage medium storing a computer program, which, when executed by a processor, can be used in the above-described multi-party cross-domain collaborative data annotation method and / or system.

[0018] Compared with the prior art, the beneficial effects of the present invention are that by setting up a modality adaptation correction unit and configuring several annotation processing unit interfaces corresponding to different modality types, the output tensors of each model are extracted and mapped to a unified data structure, label system and coordinate representation method to form a standard annotation format. This unifies the heterogeneous pre-annotation results output by different pre-trained models into a standardized format, enabling subsequent confidence assessment, task allocation and result verification modules to process multimodal data in a unified manner. This avoids the additional conversion overhead caused by format incompatibility and can effectively improve the system processing efficiency and accuracy in multimodal annotation scenarios.

[0019] Furthermore, by conducting a fine-grained capability assessment of the annotation processing units, the annotation accuracy rate under the standard annotation format is determined. This accuracy rate is then broken down and mapped to the corresponding data categories for each interface to form the interface accuracy rate. The interface adaptability is then determined based on the interface accuracy rate and the number of annotations completed per unit time. When the interface adaptability is not higher than a preset threshold, the interface is deemed unsuitable for the corresponding data category. This overcomes the shortcomings of existing technologies that only perform overall capability assessments, accurately identifying the differentiated capabilities of each annotation processing unit across different data types, achieving precise matching between tasks and capabilities, and thus effectively improving annotation accuracy.

[0020] Furthermore, during the verification annotation task, real-time availability is calculated based on the interface adaptability of the annotation processing unit and the number of assigned verification annotation tasks. Real-time availability is positively correlated with interface adaptability and negatively correlated with the current task load. Based on this, high-priority tasks are assigned to the annotation processing unit with the highest real-time availability, and similar tasks are prioritized for assignment to the annotation processing unit with the highest adaptability score. This achieves a dynamic fusion of capability assessment and load awareness, ensuring that task allocation decisions simultaneously reflect competence and idleness, resulting in good load balancing. This effectively shortens the average waiting time for annotation tasks, improves the overall system throughput, and thus significantly enhances the system processing efficiency for multimodal annotation.

[0021] Furthermore, by introducing a fatigue suppression mechanism, the continuous working time of each annotation processing unit is monitored. When the continuous working time of the corresponding device exceeds a preset threshold, the interface adaptability of that unit is automatically reduced, and the real-time availability is recalculated. This solution solves the problem that existing capability assessment methods are static and cannot reflect changes in working status. It enables the task allocation system to automatically sense and consider the fatigue state of annotation processing units, effectively suppressing the fluctuation of annotation accuracy in continuous working scenarios, and significantly improving the annotation quality of high-priority tasks. This effectively improves the system processing efficiency and accuracy in multimodal annotation scenarios.

[0022] Furthermore, by organically integrating interface adaptability assessment, real-time availability scheduling, and fatigue suppression mechanisms, a collaborative loop of capability characterization, state awareness, dynamic adjustment, and feedback optimization is formed. Interface adaptability assessment provides a fine-grained capability benchmark, real-time availability calculation combines the capability benchmark with dynamic load, and the fatigue suppression mechanism introduces a dynamic adjustment factor in the time dimension. The synergistic effect of these three mechanisms enables the cross-domain collaborative annotation system to adaptively respond to changes in task type, load fluctuations, and resource status changes, completing more annotation tasks in the same amount of time. This effectively improves the system's processing efficiency and accuracy in multimodal annotation scenarios, thereby enhancing the adaptability of data annotation. Attached Figure Description

[0023] Figure 1 This is a flowchart of the multi-party cross-domain collaborative data annotation method of the present invention; Figure 2 This is a flowchart illustrating the local pre-annotation process for a single period using a data platform, as described in this invention. Figure 3 This is a schematic diagram of the link in the multi-party cross-domain collaborative data annotation system of the present invention. Detailed Implementation

[0024] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0025] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0026] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0027] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0028] To better understand this solution, the relevant technologies are explained below: Data annotation: Data annotation is a crucial step for the effective operation of most artificial intelligence algorithms. It involves processing unprocessed data such as audio, images, text, and video into machine-readable information.

[0029] Trusted Data Space: Trusted Data Space is a distributed critical data infrastructure built on existing information networks for data aggregation, sharing, circulation, and application. It ensures the confirmation, implementation, and maintenance of data circulation protocols through systematic technical deployment, and solves security and trust issues among data element providers, users, service providers, regulators, and other entities.

[0030] Multi-party cross-domain collaborative data annotation based on trusted data space technology: Based on trusted data space, core modules such as identity authentication, access control, encrypted storage, secure transmission, auditing and clearing, and privacy computing of trusted data space are reused.

[0031] The following explains the relevant processes of this plan, including: Step S1, mark the service provider's main body for access: Multi-party cross-domain collaboration participants complete access through the Trusted Data Space's standardized connectors in accordance with the existing access specifications of the Trusted Data Space. During the access process, the Trusted Data Space's own identity authentication and permission allocation modules are reused to complete participant identity registration, qualification verification, and permission configuration.

[0032] Based on the existing identity authentication mechanism of the Trusted Data Space, a binding verification between the role of the participant and the type of annotation task is added. This verification logic is embedded in the underlying execution process of the Trusted Data Space identity authentication module, forming a linkage verification with key verification and qualification review. This ensures that the annotation service provider can only access the task type that matches its annotation capabilities, avoids abuse of permissions, and improves access efficiency.

[0033] The permissions of participating parties are configured differently based on their roles: Data provider: Has the authority to upload data, define annotation requirements, control data access, destroy raw data, and obtain pre-annotated models; The annotation service provider has the right to perform local annotation for the assigned tasks, correct pre-annotation results, and upload annotation results, but has no other permissions other than access to the original data; Spatial service provider: Has the authority to verify annotation results, review abnormal annotations, coordinate disagreements, and evaluate model performance.

[0034] Step S2, mark the task initialization configuration: Based on their own business needs, data providers create annotation task contracts in the digital contract module of the trusted data space.

[0035] On top of the existing digital contracts for data circulation and trading, digital contracts for data labeling tasks have been expanded.

[0036] The system adds automatic triggering and verification functions for annotation task permission constraints, raw data usage specifications, annotation result consistency requirements, and model update triggering clauses. It also automatically destroys the raw data after the data annotation task is completed and accepted.

[0037] When a participant violates the data usage guidelines, the contract execution engine can automatically terminate its corresponding permissions and trigger the audit module to record the exception.

[0038] The labeling task contract clearly defines the scope, priority, completion deadline, labeling rules, data usage specifications, rights and obligations of all parties, and settlement rules for the labeling task. The annotation rules include annotation format, annotation standards, annotation accuracy requirements, and divergence judgment threshold; Data usage guidelines include using it only for annotation and prohibiting its transfer.

[0039] Step S3, local de-identification processing of the dataset to be labeled: The data provider pulls a sensitive data identification model from a trusted data space, scans the original multimodal labeled data, and generates a security compliance report for the dataset. For the security compliance issues in the dataset's security compliance report, the provider selects the appropriate de-identification processing model to complete the anonymization of sensitive information and the encryption and hiding of identity information within the dataset. This ensures that the data meets compliance requirements.

[0040] The raw data includes one or more types such as images, text, audio, and structured data. Text data: The text data is identified and replaced with sensitive information such as mobile phone numbers and ID card numbers using a regular expression matching algorithm. The de-identified sensitive information is then encrypted and hidden using the SM4 national cryptographic algorithm. Image data: Pixel blurring algorithm is used to process biometric features (such as face and fingerprint), license plate number, address and other identifiable information. Coordinates of sensitive areas are encrypted and stored using the SM4 national cryptographic algorithm. Voice data: The spectrum adjustment algorithm is used to blur the identity features, and the voiceprint features are encrypted and hidden using the SM4 national cryptographic algorithm.

[0041] After processing, the dataset is encrypted and stored locally on the data provider's local machine, while the trusted data space stores only the dataset's metadata and hash check value.

[0042] Step S4, Local pre-annotation and cross-domain annotation task allocation: This invention expands and optimizes the collaborative management module and model management module of the trusted data space, reconstructs the task scheduling architecture of the collaborative management module, and adds a labeling capability evaluation submodule, a dynamic task allocation submodule, and a model scheduling submodule to replace the traditional static allocation method. The specific optimization process is as follows: Step Sy1, Pre-labeling algorithm retrieval and local execution: Based on the type of labeled dataset and the type of labeled scenario, the data provider pulls the corresponding pre-trained AI algorithm model (CNN for images, BERT for text, and Transformer for speech) from the trusted data space service platform through the model scheduling submodule, performs pre-labeling in the local environment, and generates initial pre-labeling results.

[0043] A new modal adaptation and correction unit has been added to the model output layer to convert the pre-annotation results of different modalities into a standard annotation format.

[0044] Step Sy2, Evaluation of pre-annotation results: The pre-annotation results are evaluated for quality by calculating the accuracy and confidence level. The pre-annotation results are then categorized into three levels based on confidence: high confidence (≥95%), medium confidence (70%-94%), and low confidence (<70%). High confidence results are directly used as candidate annotation results, while medium and low confidence results proceed to the manual annotation stage.

[0045] Step Sy3, Comprehensive Evaluation of Annotation Capabilities: In the trusted data space service platform, the annotation capability evaluation submodule collects historical annotation data from each annotation service provider, including indicators such as annotation type adaptability, historical annotation accuracy, annotation speed, and divergence correction response speed. It then uses the Analytic Hierarchy Process (AHP) with optimized weights for cross-domain annotation scenarios to calculate a comprehensive capability score. The weight allocation is as follows: Annotation accuracy: 40%; Annotation type adaptability: 30%; Annotation speed: 20%; Disagreement correction response speed: 10%.

[0046] The scoring range is 0-100 points, with ≥80 points being excellent, 60-79 points being satisfactory, and <60 points being unsatisfactory. Unsatisfactory results indicate that the service provider will not participate in this task allocation.

[0047] Step Sy4, Dynamic Intelligent Task Allocation: The dynamic task allocation submodule will split the dataset to be labeled according to dimensions such as data format, task type, and data volume, so as to facilitate the assignment of multiple parties to complete the labeling.

[0048] Meanwhile, based on the comprehensive capability scores of each annotation service provider, the current task load, and combined with the type, priority, and confidence level of the annotation tasks, a greedy algorithm incorporating an "annotation focus decay coefficient" is used to allocate tasks, where: High-priority, low-confidence tasks are assigned to annotation service providers with high overall capability scores and low workloads. Annotation tasks of the same type should be preferentially assigned to annotation service providers who are proficient in that type of annotation. If a labeling service provider works continuously for more than 4 hours, its overall capability score will be automatically reduced by 5% to avoid a decline in labeling quality caused by long working hours.

[0049] Step S5: Human-machine collaboration completes data annotation through a sandbox: After receiving the annotation task notification, the annotation service provider completes identity security verification through the Trusted Data Space access connector, connects to the Trusted Data Space's privacy computing platform, and completes the data annotation operation in the sandbox environment, including: Step Sz1, Sandbox Execution and Task Package Decryption: The Trusted Data Space Service Platform creates a sandbox environment and passes in the encrypted task package.

[0050] It includes anonymized annotation data, pre-annotation results, annotation requirements, and task permission information.

[0051] All annotation operations are performed in a system-isolated sandbox environment. After the task package is decrypted, functions that may lead to data leakage, such as copying, downloading, and exporting, are prohibited. A watermark of the current annotation service provider and operator is added to the page, and temporary files are stored in encrypted form throughout the process.

[0052] Step Sz2, Human-computer collaborative annotation: The annotation service provider manually corrects and supplements the pre-annotated results with medium to low confidence levels.

[0053] The system compares the pre-annotated results with the manually annotated results in real time. When the similarity is ≥95%, the pre-annotated results are used directly, and when the similarity is <95%, the manually corrected results are retained.

[0054] Step S6, Synchronization and Consistency Verification of Annotation Results: After the annotation service provider completes the annotation, it uploads the annotation results to the result storage module of the Trusted Data Space through the sandbox. The Trusted Data Space service platform then aggregates the split annotation tasks and synchronizes them to the data provider and the space service provider. During the synchronization process, the synchronization time, synchronization node, and recipient information are recorded to ensure traceability.

[0055] The trusted data space service provider uses a tiered disagreement determination method to verify the consistency of annotation results based on the annotation rules and disagreement thresholds defined by the data provider, including: Step Sf1, Verification of a single annotation result: Compare the annotation results with the annotation rules to check whether the annotation accuracy and format meet the requirements. If they do not meet the requirements, return the annotation service provider for correction. Step Sf2, consistency verification of results from multiple annotation parties: When the same batch of data is labeled by multiple labeling service providers, calculate the similarity of the labeling results.

[0056] If the similarity is greater than or equal to the divergence threshold, the result is considered consistent, and the majority of the labeled results is taken as the final result. If the similarity is less than the disagreement threshold, the space service provider initiates a review process. Through the collaborative interaction module of the trusted data space, the corresponding annotation service provider is organized to communicate online to clarify the disagreement and then re-verify. The threshold for determining disagreement is generally set at 95%.

[0057] Step Sf3, handling hierarchical branching: For core discrepancies, such as annotations that do not meet accuracy standards, they are directly returned for correction. For minor discrepancies, such as minor format deviations, the system automatically corrects them. The entire verification process is recorded by the audit module of the trusted data space, generating an unalterable audit log.

[0058] Step S7, AI pre-labeled model algorithm self-iterative update: This invention optimizes the model management module of the trusted data space according to specific scenarios, and constructs a closed-loop self-iterative mechanism of "labeling-evaluation-training-update" to achieve continuous optimization of the pre-labeled model: Step Sc1, High-quality labeled data selection: From the accepted annotation results, high-quality data with an annotation accuracy of ≥98% are selected as the dataset for incremental model training. The selection process is reviewed by the space service provider to ensure the quality of the training data.

[0059] Step Sc2, Incremental Model Training: In a privacy-preserving computing platform within a trusted data space, a federated learning framework is employed for incremental model training. During training, the original data remains locally on each data provider's machine; only updated model parameter values ​​are transmitted, and homomorphic encryption is used to ensure security during parameter transmission.

[0060] Step Sc3, Model Performance Evaluation: The performance of the newly trained model is evaluated using an independent test dataset, with evaluation metrics including pre-labeled accuracy, recall, and inference speed. If the new model outperforms the current online model, it proceeds to a phased rollout; otherwise, the training parameters are readjusted and the model is retrained.

[0061] Step Sc4, Model Gray Release and Full Update: The new model will first be released in a phased rollout to 10% of the annotation tasks, with continuous monitoring of the model's performance and annotation quality.

[0062] After 72 hours of gray-scale testing with no anomalies and stable performance, the data provider approves the update, and a full update is completed. The entire model update process is recorded, and version rollback is supported.

[0063] It should be noted that only datasets and annotation results authorized by the data provider for use in AI annotation pre-trained model algorithm sub-iteration updates will be used for model iteration; Using the aforementioned confidence level, data exceeding a preset quality threshold are selected as high-quality labeled data and set as the incremental training dataset. The original data is retained locally at the data provider's location, and only the updated values ​​of the model parameters encrypted with homomorphic encryption technology are transmitted. Incremental training is performed on the pre-trained model carried by the annotation processing unit to complete the algorithm's self-iterative update. Step S8, Task Acceptance and Data Destruction: The data provider will inspect the received annotation results, including the accuracy, completeness, and conformity of the annotation format.

[0064] Upon successful acceptance, the digital contract automatically triggers the settlement process to complete the payment settlement for all parties. If the acceptance fails, the data is returned to the auditing service provider for re-verification until it passes. Throughout the annotation process, the original annotation data, pre-annotation results, and final annotation results are stored using a dual storage method: local AES-256 encrypted storage and trusted data space SM4 encrypted backup storage, enabling real-time synchronization and verification between local and backup storage.

[0065] When local data is tampered with, it can be quickly restored using backup data, and the audit module automatically records the tampering operation.

[0066] Access permissions employ a dual authorization mechanism of roles and tasks, authorizing only users to access relevant data for the corresponding tasks, and requiring secondary authentication for access operations.

[0067] After the annotation task is completed, the data provider can, as needed, use the destruction module of the trusted data space to completely destroy the original data and intermediate process data stored locally and in backup.

[0068] The destruction process uses a combination of overwrite deletion and physical deletion to ensure that the data cannot be recovered.

[0069] The audit module audits all operations, including: The entire process of data uploading, task allocation, labeling, verification, access, and data destruction is audited, generating complete audit logs that are kept for no less than 3 years to meet compliance requirements.

[0070] In one embodiment, to meet all the requirements of the above process, the system should include: The system includes a trusted data space service platform, multi-party participant terminals, and connectors. Each component follows existing trusted data space specifications to achieve interconnectivity. The Trusted Data Space Service Platform adopts a distributed architecture, reusing core modules such as identity authentication, access control, encrypted storage, secure transmission, auditing and clearing, and privacy computing. It improves the underlying architecture and execution process of the collaborative management module, digital contract module, and model management module, and adds underlying functions such as annotation capability assessment, dynamic task allocation, multi-model pre-annotation adaptation, and model self-iteration.

[0071] The data provider terminal enables operations such as customizing annotation rules, creating tasks, accepting results, destroying raw data, and reviewing model updates.

[0072] The annotation service provider's terminal is equipped with a multi-party collaborative annotation module, which supports operations such as local pre-annotation, manual correction, uploading annotation results, and viewing annotation progress.

[0073] The spatial service provider terminal includes a result verification and synchronization module and a model self-iteration module, which support operations such as batch verification of annotation results, initiation of review processes, viewing of audit logs, and model performance evaluation.

[0074] In one embodiment, to meet all the requirements of the above process and system, the device should include the following functions and modules: This system is compatible with the Trusted Data Space. It improves the underlying architecture and execution flow of the Trusted Data Space's collaborative management module, digital contract module, and model management module, rather than simply adding application-layer modules. It reuses other core modules of the Trusted Data Space, including the following functional modules, each of which connects to its corresponding modules in the Trusted Data Space through standardized interfaces: The trusted access module for the labeled service provider is used to enable the access of participating parties. It reuses the identity authentication and permission allocation module of the trusted data space and adds a role-task binding verification unit embedded in the underlying identity authentication module, which is linked with key verification and qualification review.

[0075] The annotation contract generation module is used to receive annotation task requirements from data providers, create annotation task contracts, and synchronize them to the trusted data space digital contract module. It supports custom annotation rules and task priority settings.

[0076] The de-identification processing module has a built-in sensitive data identification and de-identification data processing module, which can automatically de-identify data for different modalities.

[0077] The pre-annotation and task allocation module is embedded in the underlying scheduling architecture of the trusted data space collaborative management module. It includes a model scheduling sub-unit, a pre-annotation evaluation sub-unit, an annotation capability evaluation sub-unit, and a dynamic task allocation sub-unit. It supports pre-annotation algorithm retrieval and local execution, comprehensive evaluation of annotation capabilities, and dynamic intelligent task allocation.

[0078] The multi-party collaborative sandbox annotation module connects to the privacy computing platform and secure transmission module of the trusted data space, providing a sandboxed annotation environment, human-computer collaborative annotation tools, cross-domain secure transmission, and real-time progress synchronization functions.

[0079] The result verification and synchronization module has a built-in consistency verification unit containing a divergence classification judgment subunit, which supports encrypted synchronization of annotation results, single result verification and consistency verification of results from multiple annotation parties.

[0080] The model self-iteration module interfaces with the model management module and privacy computing platform in the trusted data space to enable high-quality labeled data selection, federated incremental training, model performance evaluation, and canary release. The entire model update process is traceable and supports version rollback.

[0081] The security and compliance module connects to the encrypted storage, access control, and auditing modules of the trusted data space to achieve dual encrypted storage, dual authorization of roles and tasks, full-process auditing, and complete data destruction.

[0082] Please see Figure 1 The diagram shows a flowchart of the multi-party cross-domain collaborative data annotation method of the present invention. It is applied to a trusted data space implementation and, based on a trusted data space, reuses core modules such as identity authentication, access control, encrypted storage, secure transmission, auditing and clearing, and privacy computing. It includes: Step S1: Input the pre-trained model as the main body; Step S2, mark the task initialization configuration; Step S3: Local de-identification processing of the dataset to be labeled; Step S4: Local pre-annotation and cross-domain annotation task allocation; Step S5: Human-machine collaboration completes data annotation through a sandbox. Step S6: Synchronize and verify the consistency of annotation results; Step S7: The algorithm of the AI ​​pre-labeled model is updated iteratively. Step S8: Task acceptance and destruction of labeled data.

[0083] Please see Figure 2 As shown, this is a flowchart of the local pre-annotation process for a single period using a data platform according to the present invention, including: Step S41: Determine several corresponding annotation processing units based on the type and scenario of the data to be annotated; Step S42: Use the annotation processing unit to pre-annotate the data to be annotated locally, and unify the pre-annotation results of different modalities into a standard format; Step S43: Determine the confidence level of each data to be labeled based on the pre-labeling results, and divide the pre-labeling results into accepted data and re-verification data based on the confidence level threshold; Step S44: The annotation processing unit sends the verification data to the data source and performs verification annotation. Step S45: Determine the results of the re-verification annotation and determine the re-verification confidence level of each re-verification data; Step S46: Iterate through the annotation processing unit to adapt the verification confidence level; Each annotation processing unit is equipped with a corresponding pre-trained model and has several corresponding interfaces.

[0084] By setting up a modality adaptation and correction unit and configuring several annotation processing unit interfaces corresponding to different modality types, the output tensors of each model are extracted and mapped to a unified data structure, label system, and coordinate representation method to form a standard annotation format. This unifies the heterogeneous pre-annotation results output by different pre-trained models into a standardized format, enabling subsequent confidence assessment, task allocation, and result verification modules to process multimodal data in a unified manner. This avoids the additional conversion overhead caused by format incompatibility and effectively improves the system processing efficiency and accuracy in multimodal annotation scenarios.

[0085] The data platform is equipped with a modality adaptation and correction unit to unify the pre-annotation results of different modalities into a standard format; The modal adaptation and correction unit is equipped with several annotation processing unit interfaces corresponding to different modal types. This includes an image interface for image data, a text interface for text data, and a voice interface for voice data. When the corresponding annotation processing unit is called to perform pre-annotation, the modality adaptation correction unit extracts the output tensors of each model and maps them to the standard annotation format. The standard annotation format includes a unified data structure, label system, and coordinate representation method, which is used to unify the format of the pre-annotated results.

[0086] When determining the confidence level of each data point to be labeled, the data platform compares the pre-labeling results with the preset reference standards or the confidence scores output by the model itself, including: Calculate the pre-labeling accuracy of each labeled object, and generate a confidence score based on the accuracy. Based on the preset first and second confidence thresholds, the pre-labeled results are divided into three levels: Results that exceed the first confidence threshold are considered high-confidence levels and are directly output as accepted data. Results that are not higher than the first confidence threshold and are higher than the second confidence threshold are considered to have a medium confidence level. Results not exceeding the second confidence threshold are classified as low confidence level; Among them, the results of medium confidence level and low confidence level are included in the retest label; The first confidence threshold is higher than the second confidence threshold, and the standard annotation format is directly used as the input data format for confidence assessment.

[0087] During the data annotation process, the data platform continuously monitors the pre-annotation confidence performance of each annotation processing unit for different types of data; When, within a preset observation window, all online annotation processing units fail to reach the first confidence threshold for the pre-labeled data of a certain type, This type of data is determined to belong to the hard example distribution interval of the pre-trained model. Based on the statistical distribution characteristics of the pre-labeled confidence levels within the window, a dynamic first confidence threshold is calculated. While ensuring the quality of annotation, the ratio of data accepted to data verified can be adaptively adjusted. The dynamic first confidence threshold is calculated as follows: the difference between the maximum pre-labeled confidence value of this type of data within the current window and the preset pullback interval value is taken, but not lower than the second confidence threshold.

[0088] Correspondingly, when, within the preset observation window, all online annotation processing units continuously reach or exceed the first confidence threshold for the pre-annotation of a certain type of data, It has been determined that the pre-trained model has improved its pre-labeling capability for this type of data. The first confidence threshold is increased by a preset step size to improve the quality standard of the accepted data; The dynamic first confidence threshold Th_first_dynamic = max(confidence_i) - Δ Where i ∈ all pre-labeled results within the current window, Δ is the preset pullback interval value, and Th_first_dynamic ≥ Th_second.

[0089] When adjusting upwards, Th_first_new = min(Th_first_old + step, 1.0). The step value is the preset upward adjustment step size, which is usually set to 0.02 or 0.05.

[0090] For a single annotation processing unit, methods for determining its confidence level include: Determine the accuracy of standard annotation formats; The accuracy of the annotation is broken down and mapped to the data categories corresponding to each interface to form several interface accuracy rates. The interface adaptability of the annotation processing unit is determined based on the interface accuracy and the number of annotations completed per unit time. For a single interface, if the interface adaptability is not higher than the second confidence threshold, it is determined that the interface is not applicable to the data corresponding to the interface of this category.

[0091] By conducting fine-grained capability assessments of the annotation processing units, the annotation accuracy rate under the standard annotation format is determined. This accuracy rate is then broken down and mapped to the corresponding data categories for each interface to form the interface accuracy rate. The interface adaptability is then determined based on the interface accuracy rate and the number of annotations completed per unit time. When the interface adaptability is not higher than a preset threshold, the interface is deemed unsuitable for the corresponding data category. This overcomes the shortcomings of existing technologies that only perform overall capability assessments, accurately identifying the differentiated capabilities of each annotation processing unit across different data types, achieving precise matching of tasks and capabilities, and thus effectively improving annotation accuracy.

[0092] When performing re-verification annotation tasks, for the confidence level of a single pre-annotation result, tasks with a confidence level lower than the second confidence threshold are marked as high-priority tasks. Tasks that are at a high confidence level and require review are marked as ordinary priority tasks; The data platform calculates the real-time availability of each annotation processing unit based on the interface compatibility of the annotation processing unit and the corresponding number of assigned verification annotation tasks. Real-time availability is positively correlated with interface adaptability and negatively correlated with the current task load.

[0093] When performing verification annotation tasks, real-time availability is calculated based on the interface adaptability of the annotation processing unit and the number of assigned verification annotation tasks. Real-time availability is positively correlated with interface adaptability and negatively correlated with the current task load. Based on this, high-priority tasks are assigned to the annotation processing unit with the highest real-time availability, and similar tasks are prioritized for assignment to the annotation processing unit with the highest adaptability score. This achieves a dynamic fusion of capability assessment and load awareness, ensuring that task allocation decisions simultaneously reflect competence and idleness, resulting in good load balancing. This effectively shortens the average waiting time for annotation tasks, improves the overall system throughput, and thus significantly enhances the system processing efficiency for multimodal annotation.

[0094] During the iteration of the annotation processing unit, high-priority tasks are assigned to the annotation processing unit with the highest real-time availability, and... Prioritize assigning tasks of the same type to the annotation processing unit with the highest type suitability score; Monitor the continuous working time of each annotation processing unit. When the continuous working time of the corresponding device for a certain annotation processing unit exceeds the preset continuous working time threshold, In response to the event, the system automatically reduces the weight value of the interface adaptability in the real-time availability calculation model or the value of the interface adaptability itself, and recalculates the corresponding real-time availability based on the adjusted parameters. This reduces the probability of assigning new tasks to the annotation processing unit at the system level, thereby avoiding performance fluctuations or processing quality degradation caused by long-term continuous operation of the device.

[0095] The real-time availability is calculated as follows: Real-time availability = (α × interface adaptability) / (β × current task load) In the formula, α and β are dynamic adjustment coefficients, and the down-adjustment operation is to decrease the value of α.

[0096] In practice, α and β should be set according to the actual state of the corresponding equipment.

[0097] The interface compatibility calculation model is as follows: Interface adaptability_i = f(accuracy_i, number of completed tasks_i) Specifically, it can be a weighted sum or other synthetic function used to quantify static capabilities.

[0098] Real-time availability scheduling algorithm model (core): Real-time availability_j = (W_adaptability × normalized (interface adaptability_j) + W_load × normalized (1 / current load_j)) × decay factor; Wherein, the attenuation factor is ≤1.

[0099] The interface adaptability of the response annotation processing unit is automatically reduced, and its corresponding real-time availability is calculated to avoid performance fluctuations or processing quality degradation caused by long-term continuous operation of the device.

[0100] By introducing a fatigue suppression mechanism, the continuous working time of each annotation processing unit is monitored. When the continuous working time of the corresponding device exceeds a preset threshold, the interface adaptability of that unit is automatically reduced, and the real-time availability is recalculated. This solution solves the problem that existing capability assessment methods are static and cannot reflect changes in working status. It enables the task allocation system to automatically sense and consider the fatigue state of annotation processing units, effectively suppressing the fluctuation of annotation accuracy in continuous working scenarios, and significantly improving the annotation quality of high-priority tasks. This effectively improves the system processing efficiency and accuracy in multimodal annotation scenarios.

[0101] Please see Figure 3 As shown, it is a schematic diagram of the link in the multi-party cross-domain collaborative data annotation system of the present invention, including: The model scheduling unit is used to retrieve the corresponding annotation processing unit according to the data annotation type and scenario, and includes an adaptation and correction unit that unifies the pre-annotation results of different modalities into a standard format. The pre-labeling evaluation unit is used to calculate the confidence level of the pre-labeling results and classify the processing level. The capability assessment unit is used to collect historical annotation data from the annotation processing unit and generate interface adaptability using a multi-index comprehensive evaluation method. The dynamic allocation unit is used to split the dataset to be labeled and dynamically allocate it based on interface adaptability, load, task attributes and pre-labeling result confidence, using a task allocation strategy with fatigue suppression mechanism.

[0102] The specific operation methods of the pre-annotated evaluation unit include: Obtain the pre-annotated results after processing in a unified format, evaluate the quality of the pre-annotated results, and calculate the accuracy index that reflects the consistency between the pre-annotated results and the reference basis. A confidence index is generated based on the accuracy index to characterize the reliability of the pre-labeled results; The confidence index is compared with at least one preset confidence threshold. Based on the comparison result, the pre-labeled result is divided into at least two different confidence levels, and the corresponding level label is output.

[0103] By organically integrating interface adaptability assessment, real-time availability scheduling, and fatigue suppression mechanisms, a collaborative loop of capability characterization, state awareness, dynamic adjustment, and feedback optimization is formed. Interface adaptability assessment provides a fine-grained capability benchmark, real-time availability calculation combines the capability benchmark with dynamic load, and the fatigue suppression mechanism introduces a dynamic adjustment factor in the time dimension. The synergistic effect of these three mechanisms enables the cross-domain collaborative annotation system to adaptively respond to changes in task type, load fluctuations, and resource status changes, completing more annotation tasks in the same amount of time. This effectively improves the system's processing efficiency and accuracy in multimodal annotation scenarios, thereby enhancing the adaptability of data annotation.

[0104] When completing the confidence level classification, the pre-labeled evaluation unit performs verification based on the level identifier, including: If the response belongs to the first confidence level of the pre-labeled result, it will be output as a candidate labeling result and will not be sent to the review stage; For pre-labeled results belonging to the second confidence level, they are sent to the verification stage, and can be assigned corresponding task priorities according to their confidence level, which can be used to guide the selection of labeling processing units when allocating tasks in the future. The capability assessment unit records the confidence index and level label of each pre-labeled result for use during model iteration or quality control.

[0105] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0106] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-party cross-domain collaborative data annotation method, which is based on a trusted data space and reuses several modules of the trusted data space, wherein the annotation method includes at least local pre-annotation using a data platform, characterized in that, The steps for local pre-labeling for a single period using a data platform include: Based on the type and scenario of the data to be labeled, determine the corresponding labeling processing units; The annotation processing unit is used to pre-annotate the data to be annotated locally, and the pre-annotation results of different modalities are unified into a standard format; The confidence level of each data to be labeled is determined based on the pre-labeling results, and the pre-labeling results are divided into accepted data and re-verification data based on the confidence level threshold. The annotation processing unit sends the re-verification data to the data source and performs re-verification annotation; Determine the results of the retest annotations and determine the retest confidence level for each retest data; The annotation processing unit is iterated to adapt to the verification confidence level, including: Using the aforementioned confidence level, data exceeding a preset quality threshold are selected as high-quality labeled data and set as the incremental training dataset. The original data is retained locally at the data provider's location, and only the updated values ​​of the model parameters encrypted with homomorphic encryption technology are transmitted. Incremental training is performed on the pre-trained model carried by the annotation processing unit to complete the algorithm's self-iterative update. Each annotation processing unit is equipped with the corresponding pre-trained model and has several corresponding interfaces.

2. The data annotation method for multi-party cross-domain collaboration according to claim 1, characterized in that, The data platform is equipped with a modality adaptation correction unit to unify the pre-annotation results of different modalities into a standard format; The modal adaptation and correction unit is configured with several annotation processing unit interfaces corresponding to different modal types. This includes an image interface for image data, a text interface for text data, and a voice interface for voice data. When the corresponding annotation processing unit is invoked to perform pre-annotation, the modality adaptation correction unit extracts the output tensors of each model and maps them to the standard annotation format; The standard annotation format includes a unified data structure, label system, and coordinate representation method, which is used to unify the format of the pre-annotation results.

3. The data annotation method for multi-party cross-domain collaboration according to claim 2, characterized in that, When determining the confidence level of each piece of data to be labeled, the data platform compares the pre-labeling results with the preset reference standard or the confidence score output by the model itself, including: Calculate the pre-labeling accuracy of each labeled object, and generate a confidence score based on the accuracy. Based on the preset first and second confidence thresholds, the pre-labeled results are divided into three levels: Results exceeding the first confidence threshold are considered high confidence levels and are directly output as the accepted data. Results that are not higher than the first confidence threshold and are higher than the second confidence threshold are considered to have a medium confidence level. Results not exceeding the second confidence threshold are classified as low confidence level; The results of both the medium confidence level and the low confidence level are included in the retest labeling. The first confidence threshold is higher than the second confidence threshold, and the standard annotation format is directly used as the input data format for confidence assessment.

4. The data annotation method for multi-party cross-domain collaboration according to claim 3, characterized in that, For a single annotation processing unit, methods for determining its confidence level include: Determine the accuracy of the annotation in the standard annotation format; The labeled accuracy rates are broken down and mapped to the data categories corresponding to each interface to form several interface accuracy rates corresponding to each interface. The interface adaptability of the annotation processing unit is determined based on the interface accuracy and the number of annotations completed per unit time. For a single interface, if the interface adaptability is not higher than the second confidence threshold, it is determined that the interface is not applicable to the data corresponding to the interface of that category.

5. The data annotation method for multi-party cross-domain collaboration according to claim 3, characterized in that, When performing verification and annotation tasks, for the confidence level of a single pre-annotation result, tasks with a confidence level lower than the second confidence threshold are marked as high-priority tasks. Tasks that fall under the high confidence level and require review are marked as ordinary priority tasks; The data platform calculates the real-time availability of each annotation processing unit based on the interface compatibility of the annotation processing unit and the corresponding number of allocated verification annotation tasks. The real-time availability is positively correlated with the interface adaptability and negatively correlated with the current task load.

6. The multi-party cross-domain collaborative data annotation method according to claim 4 or 5, characterized in that, During the iteration of the annotation processing unit, high-priority tasks are assigned to the annotation processing unit with the highest real-time availability, and, Prioritize assigning tasks of the same type to the annotation processing unit with the highest type suitability score; Monitor the continuous working time of each annotation processing unit. When the continuous working time of the corresponding device for a certain annotation processing unit exceeds the preset continuous working time threshold, If the response exceeds the continuous working duration threshold, the weight value of the interface adaptability in the real-time availability calculation model or the value of the interface adaptability itself will be automatically reduced. And calculate the corresponding real-time availability based on the reduced weight value or interface adaptability value, in order to avoid performance fluctuations or processing quality degradation caused by long-term continuous operation of the equipment. When the adaptability of any interface fails to reach the second confidence threshold, the second confidence threshold is lowered. When the adaptability of any interface can reach the first confidence threshold, the first confidence threshold is increased.

7. A multi-party cross-domain collaborative data annotation system, which applies the method described in any one of claims 1-6, characterized in that, include: The model scheduling unit is used to retrieve the corresponding annotation processing unit according to the data annotation type and scenario, and includes an adaptation and correction unit that unifies the pre-annotation results of different modalities into a standard format. The pre-labeling evaluation unit is used to calculate the confidence level of the pre-labeling results and classify the processing level. The capability assessment unit is used to collect historical annotation data from the annotation processing unit and generate interface adaptability using a multi-index comprehensive evaluation method. The dynamic allocation unit is used to split the dataset to be labeled and dynamically allocate it based on interface adaptability, load, task attributes and pre-labeling result confidence, using a task allocation strategy with fatigue suppression mechanism.

8. The multi-party cross-domain collaborative data annotation system according to claim 7, characterized in that, The specific operation mode of the pre-labeled evaluation unit includes: Obtain the pre-annotated results after processing in a unified format, evaluate the quality of the pre-annotated results, and calculate the accuracy index that reflects the consistency between the pre-annotated results and the reference basis. Based on the accuracy index, a confidence index is generated to characterize the reliability of the pre-labeled results; The confidence index is compared with at least one preset confidence threshold. Based on the comparison result, the pre-labeled result is divided into at least two different confidence levels, and the corresponding level identifier is output.

9. The multi-party cross-domain collaborative data annotation system according to claim 8, characterized in that, When completing the confidence level classification, the pre-labeled evaluation unit performs verification based on the level identifier, including: If the response belongs to the first confidence level of the pre-labeled result, it will be output as a candidate labeling result and will not be sent to the review stage; For pre-labeled results belonging to the second confidence level, they are sent to the verification stage, and can be assigned corresponding task priorities according to their confidence level, which can be used to guide the selection of labeling processing units when allocating tasks in the future. The capability assessment unit records the confidence index and level identifier of each pre-labeled result for use during model iteration or quality control.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it can be used to perform the multi-party cross-domain collaborative data annotation method according to any one of claims 1-6.

Images