Data labeling method and device, equipment and storage medium

Abstract

The present disclosure provides a data labeling method, device and equipment and storage medium. The method is applied to a data labeling system, the data labeling system comprises a business platform and at least one labeling engine, and the method comprises the following steps: receiving to-be-labeled data through the business platform and constructing a platform data model based on the to-be-labeled data; determining a target labeling engine matched with the to-be-labeled data based on a stored mapping table through the business platform; performing pre-labeling processing on the to-be-labeled data through the business platform to obtain a platform pre-labeling result; generating a labeling configuration file matched with the target labeling engine based on historical labeling results belonging to the same task as the to-be-labeled data and the platform data model through the business platform; and performing labeling processing on the labeling configuration file based on the platform pre-labeling result through the target labeling engine to determine a target labeling result. The method effectively realizes decoupling of the business platform and the labeling engine and improves the adaptation ability and flexible expansion ability of the business platform to multiple labeling engines.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more specifically, to a data annotation method, apparatus, device, and storage medium. Background Technology

[0002] With the widespread application of artificial intelligence technology across various industries, the demand for high-quality labeled data is growing rapidly. A typical architecture for a data labeling system consists of a platform business layer that provides entry points and display functions for project management, task allocation, and personnel management, while the underlying layer relies on third-party or self-developed labeling engines to perform the actual labeling operations. The platform business layer and the underlying labeling engine are linked through key identifiers such as project number, task number, and label number.

[0003] When performing data annotation, the platform's business layer typically relies directly on the underlying annotation engine's project number, task number, label configuration, and annotation result structure to store and manage data. When the underlying annotation engine undergoes version upgrades, field changes, or needs to be replaced with another engine, the data on the platform side and the engine side often become mismatched, requiring significant investment in modifications. This tightly coupled architecture of the data annotation system severely limits the platform's business layer's adaptability and flexible scalability to multiple annotation engines. Summary of the Invention

[0004] This disclosure provides at least one data annotation method, apparatus, device, and storage medium. By establishing a business platform data model for business platform data management, and querying and calling the required target annotation engine based on a mapping table when annotation is needed, the business platform and annotation engine are effectively decoupled, thereby improving the business platform's adaptability and flexible expansion capabilities to multiple annotation engines.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] This disclosure provides a data annotation method applied to a data annotation system, the data annotation system including a business platform and at least one annotation engine; the method includes: The platform receives data to be annotated through the business platform and constructs a platform data model based on the data to be annotated. The platform data model includes project entities, task entities, file entities, annotation object entities, annotation thread entities, issue item entities, and version snapshot entities. The business platform determines the target annotation engine that matches the data to be annotated based on a stored mapping table; the mapping table is used to indicate the mapping relationship between the business platform and the annotation engine; the mapping table includes platform project number, platform file number, platform annotation object number, engine type, engine project number, engine task number, and engine annotation number; The data to be labeled is pre-labeled through the business platform to obtain the platform pre-labeling results; The business platform generates a labeling configuration file that matches the target labeling engine based on historical labeling results of the same type of task as the data to be labeled and the platform's data model. The target annotation engine performs annotation processing on the annotation configuration file based on the platform's pre-annotation results to determine the target annotation result of the data to be annotated.

[0007] In one optional implementation, the step of generating a labeling configuration file matching the target labeling engine through the business platform based on historical labeling results belonging to the same type of task as the data to be labeled and the platform's data model includes: Based on the platform's data model, the business platform determines the data modality of the data to be labeled and the labeling information of the project entities. Based on the data modality, determine the annotation control type that matches the data to be annotated; Based on the annotation label information, the annotation control metamodel corresponding to the annotation control type is configured to obtain a candidate metamodel; the annotation control metamodel includes type identifier, geometric dimension, attribute list, validation rules and rendering parameters; Based on the historical annotation results of the same type of task as the data to be annotated, the candidate meta-model is adjusted to obtain the target meta-model; Based on the target meta-model, an annotation configuration file matching the target annotation engine is generated.

[0008] In one optional implementation, the step of pre-labeling the data to be labeled through the business platform to obtain the platform pre-labeling result includes: The business platform invokes at least one intelligent agent to perform pre-labeling processing on the data to be labeled, and obtains at least one candidate pre-labeling result; different intelligent agents have different labeling functions; Each candidate pre-annotation result is format-converted to obtain a format-converted candidate pre-annotation result; the data structure of the format-converted candidate pre-annotation result is consistent with the data structure of the platform annotation object. Obtain at least one corrected pre-labeling result, wherein the corrected pre-labeling result is the result after the user corrects the candidate pre-labeling result after the format conversion; At least one of the corrected pre-annotation results is converted to a new format to obtain at least one converted corrected pre-annotation result; the data structure of the converted corrected pre-annotation result matches the annotation format of the target annotation engine. The platform pre-annotation result is obtained based on the corrected pre-annotation result after at least one format conversion.

[0009] In one optional implementation, the method further includes: Based on the label type, error type, and invoked agent, the candidate pre-labeling results and the format-converted corrected pre-labeling results are clustered to determine multiple corrected datasets; If the amount of data in any of the modified datasets reaches a preset data amount threshold, an incremental training dataset is generated based on the modified dataset, and an incremental training prompt for the agent is issued. In response to receiving an incremental training instruction for the agent, the agent corresponding to the corrected dataset is trained based on the incremental training dataset.

[0010] In one optional implementation, the method further includes: In response to a change in the labeled object entity, the change information of the labeled object entity in the business platform is transmitted to the target labeling engine; In response to a change in the annotation results stored in the target annotation engine, the engine annotation result change information of the target annotation engine is transmitted to the business platform.

[0011] In one optional implementation, each entity in the platform data model is labeled with a data version number; the method further includes: If a conflict is detected between the data version number and the update time of the target annotation engine, a conflict record is generated. The conflict record includes the platform data model of the business platform, the engine annotation result of the target annotation engine, and the difference field. The entity change information of the business platform is transmitted to the target annotation engine.

[0012] In one optional implementation, the method further includes: Receive the user's review result for the target annotation results, the review result including review passed and review failed; If the audit result indicates that the audit is not approved, an issue item entity is created, and the issue item entity is marked with a status indicating that it is pending processing. In response to detecting that the user has modified an issue item entity whose status is marked as pending, the status of the issue item entity is switched to being processed. In response to detecting that the user has submitted a correction result for an issue item entity whose status is marked as "processing", the status of the issue item entity is switched to "pending review". In response to detecting the user's confirmation of the issue item entity marked as pending review, the status of the issue item entity is switched to closed; If any of the aforementioned problem items is marked as being processed and the status has not been switched to pending review after the preset processing time limit has expired, the status of the problem item will be switched to upgraded.

[0013] According to a second aspect of this application, embodiments of this disclosure also provide a data annotation apparatus for use in a data annotation system, the data annotation system including a business platform and at least one annotation engine; the apparatus includes: The model building module is used to receive data to be labeled through the business platform and build a platform data model based on the data to be labeled. The platform data model includes project entities, task entities, file entities, labeling object entities, annotation thread entities, issue item entities, and version snapshot entities. The engine determination module is used to determine the target annotation engine that matches the data to be annotated based on the stored mapping table of the business platform; the mapping table is used to indicate the mapping relationship between the business platform and the annotation engine; the mapping table includes platform project number, platform file number, platform annotation object number, engine type, engine project number, engine task number and engine annotation number; The platform pre-labeling module is used to pre-label the data to be labeled through the business platform to obtain the platform pre-labeling result; The annotation configuration module is used to generate an annotation configuration file that matches the target annotation engine based on the historical annotation results of the data to be annotated and the platform data model through the business platform. The result determination module is used to perform annotation processing on the annotation configuration file based on the platform's pre-annotation results using the target annotation engine, and determine the target annotation result of the data to be annotated.

[0014] According to a third aspect of this application, a computer device is provided, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the data annotation method described in the first aspect are performed.

[0015] According to a fourth aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, the computer program being executed by a computer device to perform the steps of the data annotation method described in the first aspect.

[0016] The data annotation method, apparatus, device, and storage medium provided in this disclosure establish a business platform data model for data management, achieving unified and standardized management of the entire annotation business data chain. When annotation is required, the target annotation engine is invoked based on a mapping table query, effectively decoupling the business platform from the annotation engine and improving the business platform's adaptability and flexible expansion capabilities to multiple annotation engines. Thus, when it is necessary to replace or add annotation engines, only the engine adaptation layer and format converter need to be expanded, without modifying the main business process and main data structure of the business platform. This reduces the scope and cost of replacing annotation engines and accessing multiple annotation engines, and improves the stability of data annotation processing.

[0017] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure.

[0018] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram of a data annotation system provided in an embodiment of this disclosure is shown; Figure 2 A flowchart of a data annotation method provided by an embodiment of this disclosure is shown; Figure 3 This illustration shows a data annotation process provided by an embodiment of the present disclosure; Figure 4 One of the schematic diagrams of a data annotation apparatus provided in an embodiment of this disclosure is shown; Figure 5 A second schematic diagram of a data annotation device provided in an embodiment of this disclosure is shown; Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0023] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0024] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0025] All user-related data involved in the technical solutions of this application are data authorized by the user. It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, the user should be informed of the type, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and the user's authorization should be obtained.

[0026] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the computer device, application program, server, or storage medium, or other software or hardware performing the operations of this disclosed technical solution.

[0027] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the computer device.

[0028] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0029] Research has revealed that in data annotation systems, the platform's master business data and the underlying annotation engine are often tightly coupled. The platform's business layer typically relies directly on the project number, task number, label configuration, and annotation result structure of the underlying annotation engine to store and manage data. When the underlying annotation engine undergoes version upgrades, field changes, or needs to be replaced with another engine, inconsistencies can easily arise in the platform's task status, review status, issue items, version snapshots, and exported results, requiring significant investment in modifications. This tightly coupled architecture of the data annotation system severely limits the platform's business layer's adaptability and flexible scalability to multiple annotation engines.

[0030] Based on the above research, this disclosure provides a data annotation method that establishes a business platform data model to manage the data of the business platform, thereby achieving unified and standardized management of the entire data chain of the annotation business. When annotation is required, the method queries and calls the required target annotation engine based on the mapping table, effectively decoupling the business platform from the annotation engine and improving the business platform's adaptability and flexible expansion capabilities to multiple annotation engines.

[0031] To facilitate understanding of this embodiment, a data annotation method disclosed in this disclosure will first be described in detail. The data annotation method provided in this disclosure generally executes a computer device with certain computing capabilities. This computer device can be a server, which can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms. In some possible implementations, the computer device can also be a terminal device, such as user equipment (UE), mobile device, user terminal, terminal, handheld device, computing device, wearable device, etc. In other implementations, this data annotation method can be applied to an implementation environment composed of a terminal device and a server. Furthermore, this data annotation method can also be implemented by a processor calling computer-readable instructions stored in memory.

[0032] The following description, in conjunction with the accompanying drawings, illustrates a data annotation method provided by an embodiment of this disclosure.

[0033] See Figure 1 The diagram shown is a schematic representation of a data annotation system provided in an embodiment of this disclosure. Figure 1 As shown, the data annotation system 100 includes a business platform 110 and at least one annotation engine 120. In this example, N annotation engines 120 are used as an example for illustration. The data annotation system 100 also includes an engine adapter 130, which includes an engine project manager 131, an engine task manager 132, and an annotation result synchronization manager 133.

[0034] The engine adapter 130 is used to maintain a mapping table and perform bidirectional mapping between platform-side identification information and engine-side identification information, so as to realize the unified scheduling of at least one annotation engine by the business platform.

[0035] Here, the at least one annotation engine includes, for example, a rectangular bounding box annotation tool (LabelImg), an image segmentation annotation tool (LabelMe), a computer vision annotation tool (CVAT), and a visual data intelligent closed-loop annotation tool (Supervisely).

[0036] See Figure 2 The diagram shown is a flowchart of a data annotation method provided in an embodiment of this disclosure. The data annotation method is applied to, for example... Figure 1 The data annotation system shown includes a business platform and at least one annotation engine. Figure 2 As shown in the figure, the data annotation method provided in this embodiment includes steps S201 to S205, wherein: S201: Receive the data to be annotated through the business platform, and construct a platform data model based on the data to be annotated. The platform data model includes project entities, task entities, file entities, annotation object entities, annotation thread entities, issue item entities, and version snapshot entities.

[0037] Here, the platform-side projects, tasks, files, annotation objects, annotation threads, review issues, and version snapshots are used as the business truth data (i.e., master data), and task status, review status, issues, version snapshots, and export logic are uniformly built on the platform data model; the project number, task number, and annotation number of the annotation engine are stored only as mapping fields. Thus, during data processing, the platform data model is used as the business truth, and the annotation engine is used as a pluggable execution tool.

[0038] The project entity includes project ID, project name, data type, tag system configuration, creation time, and project status. The data type includes, for example, images, audio, video, text, and point clouds.

[0039] The task entity includes task number, project number, task name, assigned personnel, task status, and deadline. The task status may include, for example, pending annotation, pending review, under review, or completed.

[0040] The file entity includes file number, task number, file name, file path, file type, file size, upload time, etc.

[0041] The labeled object entity includes the labeled object number, the file number to which it belongs, the tag name, the tag color, the target type, the confidence level, the source, the generation time, the creator, and the last modifier. The source may include, for example, human or artificial intelligence (AI) models. Depending on the modality of the labeled object entity, it may also include different fields such as geometric coordinates, time range, and text range.

[0042] The annotation thread entity includes annotation number, associated annotation object number, annotation content, annotator, annotation time, and reply thread, etc. The issue item entity includes issue item number, associated labeled object number, issue type, issue description, issue status, severity level, creator, creation time, processing opinion, processor, and processing time. The issue item status includes, for example, pending processing, pending review, closed, and escalated.

[0043] The version snapshot entity includes version number, associated project number or associated task number, parent version number, stage identifier, version description, number of issues, difference description, creator, creation time, etc.

[0044] Optionally, in the platform data model, the project entity and the task entity have a one-to-many relationship, the task entity and the file entity have a one-to-many relationship, and the file entity and the annotation object entity have a one-to-many relationship; the annotation thread entity, issue item entity, and version snapshot entity all use the annotation object number or task number as the association key. The platform-side task status, review status, issue item status, version stage, and export results are all based on the platform data model, and the annotation engine only saves the external numbers and native result formats required for performing annotation processing. All entities save creation time, update time, creator, last modifier, and data version number for synchronization conflict detection and audit trail.

[0045] S202: The business platform determines the target annotation engine that matches the data to be annotated based on the stored mapping table; the mapping table is used to indicate the mapping relationship between the business platform and the annotation engine; the mapping table includes platform project number, platform file number, platform annotation object number, engine type, engine project number, engine task number, and engine annotation number.

[0046] In this embodiment of the disclosure, the engine project number, engine task number, and engine annotation number are stored in the mapping table and managed as mapping fields.

[0047] Specifically, a mapping table can be established between the business platform and the annotation engine. This mapping table includes the platform project number, platform file number, platform annotation object number, engine type, engine project number, engine task number, and engine annotation number. Here, the platform project number, platform file number, and engine type constitute a unique constraint, as do the platform annotation object number and engine annotation number.

[0048] Optionally, the mapping table may also include binding time, last synchronization time, synchronization version number, and mapping status. The mapping status may include, for example, normal, pending creation, pending synchronization, conflict, fixed, and invalid.

[0049] For example, the platform file number is PF_001, the engine project number is EP_10001, the engine task number is ET_20001, the engine annotation number is EA_30001, the engine type is CVAT, the binding time is 2026-01-01, and the last synchronization time is 2026-06-20.

[0050] In this embodiment of the disclosure, all business logic of the business platform, such as task status transition, review process, version management, data export, etc., is based on the platform data model. Only when it is necessary to call the annotation engine for annotation processing, the target annotation engine that matches the data to be annotated is determined by querying the mapping table, which effectively decouples the business platform from the underlying annotation engine.

[0051] S203: The data to be labeled is pre-labeled through the business platform to obtain the platform pre-labeling result.

[0052] See you again Figure 1 The business platform 110 can call the intelligent agent 140 to perform pre-labeling processing on the data to be labeled, and obtain the platform pre-labeling result, thereby reducing the cost of manual labeling.

[0053] In some possible implementations, the step of pre-labeling the data to be labeled through the business platform to obtain the platform pre-labeling result includes: The business platform invokes at least one intelligent agent to perform pre-labeling processing on the data to be labeled, and obtains at least one candidate pre-labeling result; different intelligent agents have different labeling functions; Each candidate pre-annotation result is format-converted to obtain a format-converted candidate pre-annotation result; the data structure of the format-converted candidate pre-annotation result is consistent with the data structure of the platform annotation object. Obtain at least one corrected pre-labeling result, wherein the corrected pre-labeling result is the result after the user corrects the candidate pre-labeling result after the format conversion; At least one of the corrected pre-annotation results is converted to a new format to obtain at least one converted corrected pre-annotation result; the data structure of the converted corrected pre-annotation result matches the annotation format of the target annotation engine. The platform pre-annotation result is obtained based on the corrected pre-annotation result after at least one format conversion.

[0054] In the above steps, the task context can be read through the business platform; the task context includes file type, tag system, task description, historical examples, target output mode (Schema), and quality constraints. A suitable agent can be selected from multiple agents based on the task context.

[0055] The annotation functions of multiple intelligent agents include artificial intelligence (AI) capabilities such as optical character recognition (OCR), open vocabulary object detection, speech recognition, video analysis, and large language models (e.g., multimodal models, large text models, etc.).

[0056] Among them, OCR recognition is used to detect and recognize text in images; open vocabulary object detection is used to detect objects based on zero or few samples of an open vocabulary; speech recognition is used to process audio data into speech-to-text; video analysis is used to extract keyframes from videos and perform content analysis; multimodal model is a multimodal understanding model that combines images, text, etc.; and large text model is used to perform semantic understanding and entity extraction of text data.

[0057] After selecting a suitable agent, an agent invocation prompt can be generated for the selected agent. This prompt includes label definitions, output fields, and coordinate or time format requirements. Based on the prompt, the selected agent is invoked to perform pre-labeling processing on the data to be labeled, obtaining candidate pre-labeling results. It can be understood that at least one agent is invoked, and correspondingly, at least one candidate pre-labeling result is obtained.

[0058] Optionally, the candidate pre-labeling results can be validated to check field completeness, coordinate validity, label validity, and confidence range, etc., to obtain a validation result for the candidate pre-labeling results. If the validation result indicates that the candidate pre-labeling result has passed validation, subsequent processing can be performed. If the validation result indicates that the candidate pre-labeling result has failed validation, retry, switching agents, or reverting to manual annotation can be triggered.

[0059] Optionally, the data annotation system may include an agent invocation adaptation layer. This layer encapsulates the invocation methods, parameter formats, and authentication mechanisms for different agents and provides an invocation interface for the agents. Thus, agents can be invoked through this interface. When adding a new agent, it only needs to be registered in the adaptation layer to connect.

[0060] In traditional data annotation methods, the data formats of the output results from agents with different annotation functions vary, including but not limited to JavaScript Object Notation (JSON), Extensible Markup Language (XML), plain text, and proprietary binary formats. Traditional methods lack the ability to directly convert these heterogeneous outputs into unified annotation objects that are editable, synchronizeable, reviewable, and traceable within the annotation engine. This results in a disconnect between the agent's pre-annotation results and the manual review process, making it difficult to manually review and correct the agent's pre-annotation results.

[0061] In this embodiment, each candidate pre-labeling result is format-converted to obtain a format-converted candidate pre-labeling result. The data structure of the format-converted candidate pre-labeling result is consistent with the data structure of the platform labeling object, thereby unifying the heterogeneous outputs of different intelligent agents into the data structure of the platform labeling object.

[0062] The platform's labeled object data structure combines unified fields and modal extension fields. The unified fields include: labeled object number, file number, label code, label color, label name, target type, geometric coordinates / range, confidence level, source type, source model, generation time, original response summary, and data version number. The modal extension fields include: for image modalities, bounding boxes, polygons, keypoints, or masks; for text modalities, start and end character offsets and entity text; for audio modalities, start and end timestamps and transcribed text; for video modalities, time periods, keyframe numbers, and intra-frame bounding boxes; and for point cloud modalities, 3D bounding boxes, point set indexes, or segmented regions.

[0063] Wherein, the labeled object number is a unique identifier generated by the business platform, and the labeled object number is AO_20260620_001 for example. The label name is the name of the labeled category, and the label name is "vehicle" for example. The label color is a visual color, and the label color is #FF0000 for example. The target type is the labeled geometry type, and the target type is RECTANGLE / POLYGON / TIMESEGMENT for example. The geometric coordinates / range are used to indicate the labeled location information, and different fields are used for different modalities. For example, the geometric coordinates / range is {"x":100,"y":50,"w":200,"h":150}. The confidence score is the confidence score of the agent, and the confidence score is 0.87 for example. The source type is the name of the agent that generated the candidate pre-labeled result, and the source type is YOLOv8 for example. The generation time is the timestamp of the candidate pre-labeling result generation, for example, 2026-06-20T10:30:00Z. The original response summary is a summary of the key information originally output by the agent, for example, "raw":"...".

[0064] When converting the format of the candidate pre-labeling results, specifically, a converter matched with the agent that generated the candidate pre-labeling results can be used to convert the format of the candidate pre-labeling results. For example, the converter matched with the agent that generated the candidate pre-labeling results can parse the candidate pre-labeling results to obtain target parsing data, and generate the format-converted candidate pre-labeling results based on the target parsing data. For example, for an agent whose annotation function is open vocabulary object detection, the target parsing data includes bounding box coordinates, category labels, and confidence scores, etc.; for an agent whose annotation function is OCR, the target parsing data includes text box coordinates and recognized text, etc.; for an agent whose annotation function is speech recognition, the target parsing data includes timestamps and transcribed text fragments, etc.; for an agent whose annotation function is a large language model, the target parsing data includes entity location and type.

[0065] After generating the format-converted candidate pre-annotation results, these results can be displayed to users (e.g., annotators), for example, through the workbench interface of the business platform. Users can then correct these results and obtain the corrected pre-annotation results for each format-converted candidate pre-annotation. These corrections include, for example, confirming, deleting, adjusting the position of, modifying labels, and completing missing labels on the format-converted candidate pre-annotation results.

[0066] When converting the format of the corrected pre-annotation results, specifically, a format converter matching the target annotation engine can be used to convert the format of the corrected pre-annotation results to obtain a format-converted corrected pre-annotation result that matches the annotation format of the target annotation engine. For example, for CVAT, the format-converted corrected pre-annotation result is in XML format; for LabelImg, the format-converted corrected pre-annotation result is in Pascal VOC format. Optionally, field mapping and data type adaptation processing are performed during the format conversion process of the corrected pre-annotation results.

[0067] The platform pre-annotation result is obtained based on the corrected pre-annotation result after at least one format conversion.

[0068] In some possible implementations, upon obtaining the platform's pre-annotation results, multiple format-converted corrected pre-annotation results (from different agents or manually added) under the same file are merged. Optionally, duplicate annotation numbers can be automatically cleaned up during the merging process to ensure that each annotation object has only one corresponding annotation record. Optionally, the merged records can be saved to the mapping table, making the association between annotation objects and annotation numbers clearly traceable. In this way, by merging and deduplicating the format-converted corrected pre-annotation results, the consistency of annotation results between the business platform and the annotation engine is ensured, avoiding risks caused by data inconsistency.

[0069] In some possible implementations, when the data to be labeled is pre-labeled by multiple agents, the confidence-weighted fusion of the modified pre-labeling results after multiple format conversions can be performed to obtain the platform pre-labeling results.

[0070] Optionally, confidence-weighted fusion processing can be performed separately according to modality. For image detection boxes, Intersection over Union (IoU) can be used to determine whether they point to the same target; for text entities, character interval overlap rate and entity type consistency can be used to determine whether they point to the same target; for audio and video clips, time interval overlap rate can be used to determine whether they point to the same target; for video target trajectories, keyframe IoU and time overlap rate can be used to jointly determine whether they point to the same target; for point cloud targets, 3D IoU or point set overlap rate can be used to determine whether they point to the same target. The format-converted corrected pre-labeling results that point to the same target are weighted and fused according to the corresponding corrected confidence to obtain the platform pre-labeling results. Optionally, a fusion record can be generated, which includes the source agent, the original confidence corresponding to the candidate pre-labeling result, the corrected confidence corresponding to the corrected pre-labeling result, the fusion weight, and the merged format-converted corrected pre-labeling results.

[0071] For example, taking image detection boxes as an example, for detection boxes generated by different agents in the same location area, the IoU between every two detection boxes is calculated; two detection boxes with an IoU greater than a preset Intersection over Union (IoU) threshold are identified as pointing to the same target. The preset IoU threshold can be determined according to actual fusion needs and is not specifically limited here. For example, the preset IoU threshold is 0.5. For multiple detection boxes pointing to the same target, the positions of the fused detection boxes are determined by weighting their corresponding corrected confidence scores. The detection box with the highest corrected confidence score is selected as the main candidate box.

[0072] In this way, confidence-weighted fusion processing can help improve the annotation quality in multi-agent collaborative scenarios.

[0073] Once the platform's pre-annotation results are obtained, they can be written back to the target annotation engine through a synchronization mechanism that combines event-driven and timed verification.

[0074] Optionally, after synchronizing the platform pre-annotation results to the target annotation engine, a consistency check can be performed on the business platform and the target annotation engine. Specifically, the number of annotated objects on the business platform can be compared with the number of annotations on the target annotation engine, and the content of the platform pre-annotation results stored on the business platform can be compared with the content of the platform pre-annotation results stored on the target annotation engine. If the quantities are inconsistent or the content does not match, an alarm is triggered and timely repair is performed. After the repair is completed, the synchronization of the platform pre-annotation results is re-executed, and the platform pre-annotation results are transmitted from the business platform to the target annotation engine until the platform pre-annotation results stored on the business platform are consistent with the platform pre-annotation results stored on the target annotation engine.

[0075] In this way, by calling at least one agent to pre-annotate the data to be labeled, it can adapt to diverse pre-annotation needs; by converting each candidate pre-annotation result into a format consistent with the data structure of the platform's labeled objects, the output format differences between different agents are eliminated, making it easy for users to directly correct and adjust the pre-annotation results, avoiding the process of manually re-entering or manually converting the output results of heterogeneous agents; by converting the corrected pre-annotation results into a format that matches the labeling format of the target labeling engine and integrating them to obtain the platform's pre-annotation results, it is possible to achieve smooth flow of pre-annotated data between the platform side and the engine side, reduce the cost of manual format processing, and improve labeling efficiency.

[0076] In some possible implementations, the method further includes: Based on the label type, error type, and invoked agent, the candidate pre-labeling results and the format-converted corrected pre-labeling results are clustered to determine multiple corrected datasets; If the amount of data in any of the modified datasets reaches a preset data amount threshold, an incremental training dataset is generated based on the modified dataset, and an incremental training prompt for the agent is issued. In response to receiving an incremental training instruction for the agent, the agent corresponding to the corrected dataset is trained based on the incremental training dataset.

[0077] Here, after the user corrects the candidate pre-labeling results after the format conversion, a correction record can be generated. The correction record includes the candidate pre-labeling results after the format conversion before correction, the corrected pre-labeling results, the corrected fields, the reason for correction, the person making the correction, and the correction time.

[0078] In this step, the candidate pre-labeling results and the format-converted corrected pre-labeling results can be clustered according to label type, error type, and the source model of the candidate pre-labeling results to determine multiple corrected datasets. It can be understood that each corrected dataset includes candidate pre-labeling results and format-converted corrected pre-labeling results of the same type (same label type and error type).

[0079] If the amount of data in any of the corrected datasets reaches a preset data size threshold, an incremental training dataset is generated based on the corrected dataset, and incremental training prompts for the agent are issued. The preset data size threshold can be set according to actual incremental training needs and is not specifically limited here. For example, the preset data size threshold is 100 records.

[0080] In response to receiving an incremental training instruction for the agent, the agent corresponding to the corrected dataset is trained based on the incremental training dataset to obtain an optimized agent. Optionally, the optimized agent needs to be evaluated on a validation set before being put into use. Optionally, the unoptimized agent can be retained to support rollback.

[0081] In this way, the corrected pre-labeling results can be effectively fed back to the corresponding agent's capability optimization, forming a positive cycle from labeling correction to agent performance improvement, effectively improving the accuracy of agent pre-labeling, and thus improving the overall labeling efficiency.

[0082] S204: Based on the historical annotation results of the data to be annotated belonging to the same type of task as the data to be annotated and the platform data model, the business platform generates an annotation configuration file that matches the target annotation engine.

[0083] In traditional data annotation methods, multimodal annotation templates are difficult to adapt automatically. Because the annotation controls, label fields, geometric structures, and temporal structures corresponding to different modalities of data, such as images, audio, video, text, and point clouds, are fundamentally different, traditional methods typically require manual configuration of annotation templates for each modality, failing to automatically generate executable annotation configurations. This results in low deployment efficiency in multimodal annotation scenarios and an inability to quickly respond to diverse annotation needs.

[0084] In this embodiment of the disclosure, the business platform can generate a labeling configuration file that matches the target labeling engine based on historical labeling results of the same type of task as the data to be labeled and the platform's data model, thereby realizing the automatic generation of labeling configuration, eliminating manual configuration costs, and effectively solving the above-mentioned technical problems.

[0085] In some possible implementations, the step of generating a labeling configuration file matching the target labeling engine based on historical labeling results belonging to the same type of task as the data to be labeled and the platform's data model through the business platform includes: Based on the platform's data model, the business platform determines the data modality of the data to be labeled and the labeling information of the project entities. Based on the data modality, determine the annotation control type that matches the data to be annotated; Based on the annotation label information, the annotation control metamodel corresponding to the annotation control type is configured to obtain a candidate metamodel; the annotation control metamodel includes type identifier, geometric dimension, attribute list, validation rules and rendering parameters; Based on the historical annotation results of the same type of task as the data to be annotated, the candidate meta-model is adjusted to obtain the target meta-model; Based on the target meta-model, an annotation configuration file matching the target annotation engine is generated.

[0086] In the above steps, the business platform determines the data modality of the data to be labeled and the label information of the project entities based on the platform's data model. The data modality of the data to be labeled includes, for example, images, audio, video, text, point clouds, etc. The label information includes label name, label level, label color, etc.

[0087] Based on the data modality and the modality control mapping table, the annotation control type matching the data to be annotated is determined. The modality control mapping table is used to map the data modality to the annotation control type. Specifically, the modality control mapping table is shown in Table 1 below: Table 1

[0088] Based on the annotation information, label options, label colors, hierarchical paths, and required attributes are generated. Here, each annotation control type is based on a corresponding annotation control metamodel description. Based on the label options, label colors, hierarchical paths, and required attributes, the annotation control metamodel corresponding to the annotation control type is configured. For example, the geometric dimensions, validation rules, and rendering parameters of the annotation control metamodel corresponding to the annotation control type can be filled in to obtain a candidate metamodel.

[0089] The annotation control metamodel includes a type identifier, geometric dimensions, an attribute list, validation rules, and rendering parameters. The type identifier indicates the unique type code of the annotation control, such as RECTANGLE, POLYGON, TIMESEGMENT, etc. The geometric dimensions specifically refer to the geometric parameter dimensions required by the annotation control; for example, a rectangle represents 4-dimensional coordinates, a polygon represents a sequence of N points, and a time segment represents the start and end times. The attribute list includes configurable attribute fields for the annotation control, such as label name, label color, whether it is required, and whether multiple selections are allowed. The validation rules specifically define the validity rules for the control values, such as coordinates must be within the image range and time cannot be negative. The rendering parameters specifically refer to the visualization parameters of the annotation control in the annotation workbench, such as color, line width, and transparency.

[0090] Based on the project number, tag system version, file type, task domain, and annotation rule template, historical annotation results belonging to the same task as the data to be annotated are determined. If there is only one historical annotation result belonging to the same task as the data to be annotated, the candidate meta-model can be directly adjusted based on the historical annotation result belonging to the same task as the data to be annotated to obtain the target meta-model.

[0091] When there are multiple historical annotation results belonging to the same task as the data to be annotated, the tag usage frequency, tag co-occurrence frequency, common attribute values, and correction records of each historical annotation result can be statistically analyzed. Tags or attributes that rank highly and have an approval rate higher than a preset threshold can be selected as recommended configuration items and written into the configuration as recommended items, without overriding tags and validation rules explicitly set by the project administrator. The preset threshold can be determined according to the actual annotation configuration needs and is not specifically limited here.

[0092] Optionally, the distribution of tags in already labeled files under the same task can be automatically statistically analyzed and a tag heatmap can be generated. A preset number of tags that belong to the same task as the data to be labeled and have the highest usage frequency can be identified as the top-ranked tags for labelers to quickly select. The preset number can be determined according to actual labeling configuration needs and is not specifically limited here.

[0093] Optionally, the annotation records of the same file at different time points can be automatically merged and deduplicated.

[0094] Optionally, for annotation controls of new modalities, extensions can be achieved by registering new control types in the metamodel without modifying the core code.

[0095] Optionally, subtype controls can inherit the basic properties of their parent type controls. For example, an audio transcription control can inherit the basic properties of a text entity control and add an additional timeline property. Furthermore, based on the annotation configuration requirements, parameters can be overridden on top of the inherited basic properties of the parent type control, which helps reduce repetitive configuration processing.

[0096] Based on the target meta-model, a labeling configuration file matching the target labeling engine is generated, which is to say, an executable labeling configuration file for the target labeling engine is obtained. Optionally, the configuration version number of the labeling configuration file can be saved.

[0097] In this way, the matching annotation control type can be determined based on the data modality, and the annotation control metamodel can be configured according to the annotation label information and historical annotation results, so as to realize the automatic generation of annotation configuration, effectively reducing the workload of manually configuring annotation templates, while taking into account the compatibility between the annotation configuration file and the target annotation engine, and can meet the annotation needs of different data modalities. For multimodal annotation projects, the annotation configuration time can be compressed from hours to minutes.

[0098] S205: The target annotation engine performs annotation processing on the annotation configuration file based on the platform's pre-annotation results to determine the target annotation result of the data to be annotated.

[0099] In this step, after obtaining the annotation configuration file generated by the business platform, the engine project matching the annotation configuration file can be determined by the target annotation engine; under the engine project matching the annotation configuration file, an engine task matching the annotation configuration file is generated; by processing the engine task matching the annotation configuration file, the annotation configuration file is annotated based on the platform's pre-annotation results to determine the target annotation result of the data to be annotated.

[0100] Here, engine projects can be created or updated according to the type of the annotation configuration file. Specifically, based on the type of the annotation configuration file, it can be determined whether a corresponding engine project exists in the target annotation engine. If no corresponding engine project exists, the Application Programming Interface (API) of the target annotation engine is called to create a new engine project, and the created engine project is identified as the engine project matching the annotation configuration file. Optionally, the engine project number of the created engine project can be saved to the mapping table. If a corresponding engine project exists, the engine project and its engine project number can be directly reused.

[0101] Under an engine project matching the annotation configuration file, a platform file can be generated based on the data to be annotated and the annotation configuration file. The platform file is then converted to a new format, matching the task format recognizable by the target annotation engine. The target annotation engine's import interface is then called to import the converted file, generating an engine task matching the annotation configuration file. Optionally, the annotation configuration file number can be bound to the engine task number, and this binding relationship can be saved.

[0102] In some possible implementations, the method further includes: In response to a change in the labeled object entity, the change information of the labeled object entity in the business platform is transmitted to the target labeling engine; In response to a change in the annotation results stored in the target annotation engine, the engine annotation result change information of the target annotation engine is transmitted to the business platform.

[0103] Optionally, the annotation object entity on the business platform and the annotation results stored in the target annotation engine can be periodically checked to see if they have changed. The detection frequency can be set according to the actual needs of bidirectional synchronization, and no specific limitation is made here.

[0104] Alternatively, an event-driven approach can be used, detecting whether the labeled object entity and the labeled results stored in the target labeling engine have changed in response to the detection of a target event. The target event can be set according to actual bidirectional synchronization needs, and is not specifically limited here.

[0105] Here, transmitting change information from the platform side to the engine side is called forward synchronization; transmitting change information from the engine side to the platform side is called reverse synchronization.

[0106] Specifically, in response to changes in the labeled object entity, such as when the labeled object entity is added, modified, or deleted, the change information of the labeled object entity on the business platform can be determined. This change information may include the labeled object number, data version number, and operation type of the changed labeled object entity. The corresponding engine number can be queried according to the mapping table, and the corresponding engine interface can be called through the engine adapter based on the engine number, writing an idempotent key to synchronize the labeled object entity change information to the target labeling engine. The idempotent key can serve as an identifier during information synchronization, and includes the platform labeled object number and a synchronization timestamp. Optionally, when the business platform repeatedly initiates synchronization with the same idempotent key, the engine side can ignore the repeated synchronization operation to avoid multiple synchronizations of the same labeled object entity change information, reducing redundant synchronization data. Here, when calling the engine interface fails, a preset number of retries can be performed, and the reason for the failure can be recorded. The preset number of retries can be set according to actual bidirectional synchronization needs and is not specifically limited here.

[0107] In response to a change in the annotation results stored in the target annotation engine, such as when a user modifies the annotation results stored in the target annotation engine through the engine interface, the engine annotation result change information of the target annotation engine can be determined and transmitted to the business platform.

[0108] In this way, by transmitting change information from the business platform to the target annotation engine and vice versa, bidirectional synchronization between the business platform and the annotation engine is achieved. This ensures data consistency and state coordination between the platform and the engine, avoids annotation deviations caused by data asynchrony, and helps maintain stable data annotation execution under the decoupled architecture of the platform and the engine.

[0109] In some possible implementations, the various entities included in the platform data model are labeled with data version numbers; the method further includes: If a conflict is detected between the data version number and the update time of the target annotation engine, a conflict record is generated. The conflict record includes the platform data model of the business platform, the engine annotation result of the target annotation engine, and the difference field. The entity change information of the business platform is transmitted to the target annotation engine.

[0110] In the above steps, it is possible to periodically check whether there is a conflict between the data version number and the update time of the target annotation engine, that is, to check whether there is a conflict between the relevant information of the same annotation object on both the business platform and the annotation engine. After the annotation object on the same platform is synchronized on the business platform side and the target annotation engine side, if the data version number of the business platform changes compared to the previous synchronized version and the update time of the target annotation engine is later than the previous synchronization time, it can be determined that a conflict exists.

[0111] When a conflict is identified, a conflict record can be generated, and the entity change information of the business platform can be transmitted to the target annotation engine, so that the business platform side data can be used as the business truth value for synchronization.

[0112] Optionally, conflict resolution prompts can be issued, along with conflict resolution suggestions, such as retaining the platform version, using the engine version, or merging at the field level. Conflict resolution is performed according to user instructions, and the conflict resolution results are written to the conflict audit log.

[0113] This allows for timely identification of data version conflicts between the platform and the engine, saving conflict records for traceability and verification, and synchronizing entity change information from the business platform to the engine side. This effectively reduces the risk of data deviation caused by version inconsistencies and improves the accuracy and traceability of data synchronization.

[0114] In some possible implementations, the data version number on the business platform side can be periodically compared with the update time of the target annotation engine, and a reverse synchronization or conflict detection process can be triggered when a difference is found.

[0115] In some possible implementations, anomaly detection can also be performed on the mapping table. Specifically, this anomaly detection can involve consistency checks, comparing the number of tasks on the platform side with the number of tasks on the engine side to detect the existence of isolated or ghost tasks. An isolated task refers to a task that exists only on the engine side but has no corresponding platform task mapping relationship; the mapping table does not contain an engine task with a matching platform number. A ghost task refers to an invalid task whose platform side still retains a valid task record, but the corresponding engine task has been deleted, rendering the mapping relationship invalid. In addition to platform side anomaly detection, number conflict detection can also be performed to check for anomalies where two different platform files are mapped to the same engine task number. If anomalies are detected in the mapping table, remapping can be performed, regenerating the mapping table and marking it as repaired. A mapping table change log is also recorded for manual auditing. Thus, by detecting anomalies in the mapping table and automatically repairing them when they occur, the data consistency of the data annotation system can be ensured during long-term operation.

[0116] In traditional data annotation methods, the review results of various annotations are stored in a scattered manner or only logged, lacking a correlation around the same labeled object. Especially in scenarios that require data auditing, reproducible model training, and quality proof of dataset delivery, the source, modification process, and quality status of the annotation results are difficult to trace and prove.

[0117] In this embodiment of the disclosure, the status management of problematic entities that fail the review is refined, thereby providing records of the source, modification history and quality status of the target annotation results, which can meet the requirements of scenarios such as data auditing, reproducible model training and quality proof of dataset delivery.

[0118] In some possible implementations, the method further includes: Receive the user's review result for the target annotation results, the review result including review passed and review failed; If the audit result indicates that the audit is not approved, an issue item entity is created, and the issue item entity is marked with a status indicating that it is pending processing. In response to detecting that the user has modified an issue item entity whose status is marked as pending, the status of the issue item entity is switched to being processed. In response to detecting that the user has submitted a correction result for an issue item entity whose status is marked as "processing", the status of the issue item entity is switched to "pending review". In response to detecting the user's confirmation of the issue item entity marked as pending review, the status of the issue item entity is switched to closed; If any of the aforementioned problem items is marked as being processed and the status has not been switched to pending review after the preset processing time limit has expired, the status of the problem item will be switched to upgraded.

[0119] In the above steps, users (such as reviewers) can review the target annotation results. If the review result indicates that the review fails, an issue item entity can be created according to the issue type. The issue types include, for example, missing annotations, incorrect annotations, inaccurate boundaries, and incorrect labels. The issue item entity includes the associated annotation object number, issue type, issue description, issue severity level (e.g., warning / moderate / serious), suggested correction plan, creator, and creation time.

[0120] The status markers for issue items include five categories: Pending, Processing, Pending Review, Closed, and Escalated. After an issue item is created, its status marker is Pending. After a user modifies the issue item, its status marker changes to Processing. After a user submits a correction, its status marker changes to Pending Review. After a user confirms the issue is resolved, its status marker changes to Closed. For issue items that are still in the Processing stage and have not entered the Pending Review stage after a preset processing time limit, their status marker is automatically switched to Escalated and their severity level increased. This ensures that the issue is not left unresolved indefinitely.

[0121] The preset processing time limit can be set according to actual processing needs, and is not specifically limited here. For example, the preset processing time limit is 3 days.

[0122] Optionally, a status mark switching record can be generated each time the status mark is switched. The status mark switching record may include the processor, the processing time, the status mark before the switch, the status mark after the switch, and the processing opinion.

[0123] Optionally, if a target number of problem item entities are created for the same target annotation result, it indicates that the target annotation result has multiple problems. In this case, an annotation alarm can be issued to prompt the user to re-annotate. The specific target number can be set according to the actual data annotation needs and is not specifically limited here. For example, the target number is 3.

[0124] This enables full-process status management of annotation review, clearly tracks the processing progress of problematic entities, ensures timely follow-up and handling of issues, improves the standardization and traceability of the annotation review process, and effectively guarantees the quality of annotation results.

[0125] In some possible implementations, incremental version snapshots can be generated based on the issue entity and change history, and version replay and version rollback conflict detection can be performed.

[0126] Specifically, the version snapshot adopts an incremental storage structure. The version snapshot only stores operation change records relative to the parent version, rather than fully copying the labeled data. The version snapshot includes version number, parent version number, task number (or project number), stage identifier (e.g., initial labeling, first review, second review, final review, etc.), creator, creation time, issue count statistics, difference description, difference summary, and change record pointer. The change record saves add, modify, and delete operations at the labeled object number granularity; modification operations save the field name, the value before the change, and the value after the change.

[0127] When performing version replay, change records are read from the target version along the parent version chain and the annotation results are restored in chronological order.

[0128] When performing a version rollback, the system checks whether any marked objects have been added or deleted between the target historical version and the current version. If so, a conflict list is generated, prompting the user to choose a handling method, such as keeping, deleting, or merging. This conflict detection during version rollback helps improve the security of the rollback process.

[0129] Optionally, embodiments of this disclosure can visualize version differences. Specifically, in response to a user's need to view version snapshots, multiple version snapshots can be displayed simultaneously. Geometric changes (e.g., frame displacement, size changes, label changes, etc.) of the same labeled object in each version snapshot can be highlighted, for example, by using a highlighting method. In this way, through incremental storage and difference visualization of version snapshots, the complete modification trajectory of each labeled object from generation to final delivery can be traced, improving the traceability of annotations.

[0130] For a clearer illustration of the data annotation process, see [link to documentation]. Figure 3 This is a schematic diagram illustrating a data annotation process provided in an embodiment of this disclosure. Figure 3 As shown, a platform data model can be constructed based on the data to be labeled; engine mapping can be performed through a mapping table to determine the target labeling engine that matches the data to be labeled; pre-labeling processing can be performed on the data to be labeled to obtain the platform pre-labeling results; a labeling configuration file matching the target labeling engine can be generated; and the target labeling engine can perform labeling processing on the labeling configuration file based on the platform pre-labeling results to obtain the target labeling results. Specific steps are described in the foregoing embodiments and will not be repeated here.

[0131] The data annotation method provided in this disclosure establishes a business platform data model for data management, achieving unified and standardized management of the entire annotation business data chain. When annotation is required, the method queries and calls the required target annotation engine based on a mapping table, effectively decoupling the business platform from the annotation engine and improving the business platform's adaptability and flexible expansion capabilities to multiple annotation engines. Thus, when it is necessary to replace or add an annotation engine, only the engine adaptation layer and format converter need to be extended, without modifying the main business process and main data structure of the business platform. This reduces the scope and cost of replacing annotation engines and accessing multiple annotation engines, and improves the stability of data annotation processing.

[0132] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0133] Based on the same inventive concept, this disclosure also provides a data annotation device corresponding to the data annotation method. Since the principle of the data annotation device in this disclosure is similar to that of the data annotation method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0134] Please see Figure 4 and Figure 5 , Figure 4 This is one of the schematic diagrams of a data annotation device provided in an embodiment of this disclosure. Figure 5 This is a second schematic diagram of a data annotation device provided in an embodiment of this disclosure. The data annotation device is applied to a data annotation system, which includes a business platform and at least one annotation engine. Figure 4 As shown in the illustration, the data annotation apparatus 400 provided in this embodiment includes: The model building module 401 is used to receive the data to be labeled through the business platform and build a platform data model based on the data to be labeled. The platform data model includes project entities, task entities, file entities, labeling object entities, annotation thread entities, issue item entities, and version snapshot entities. The engine determination module 402 is used to determine the target annotation engine that matches the data to be annotated based on a stored mapping table of the business platform; the mapping table is used to indicate the mapping relationship between the business platform and the annotation engine; the mapping table includes platform project number, platform file number, platform annotation object number, engine type, engine project number, engine task number and engine annotation number; The platform pre-labeling module 403 is used to pre-label the data to be labeled through the business platform to obtain the platform pre-labeling result; The annotation configuration module 404 is used to generate an annotation configuration file that matches the target annotation engine based on the historical annotation results of the data to be annotated and the platform data model through the business platform. The result determination module 405 is used to perform annotation processing on the annotation configuration file based on the platform's pre-annotation results by the target annotation engine, and determine the target annotation result of the data to be annotated.

[0135] In one optional implementation, the annotation configuration module 404 is specifically used for: Based on the platform's data model, the business platform determines the data modality of the data to be labeled and the labeling information of the project entities. Based on the data modality, determine the annotation control type that matches the data to be annotated; Based on the annotation label information, the annotation control metamodel corresponding to the annotation control type is configured to obtain a candidate metamodel; the annotation control metamodel includes type identifier, geometric dimension, attribute list, validation rules and rendering parameters; Based on the historical annotation results of the same type of task as the data to be annotated, the candidate meta-model is adjusted to obtain the target meta-model; Based on the target meta-model, an annotation configuration file matching the target annotation engine is generated.

[0136] In one optional implementation, the platform pre-annotation module 403 is specifically used for: The business platform invokes at least one intelligent agent to perform pre-labeling processing on the data to be labeled, and obtains at least one candidate pre-labeling result; different intelligent agents have different labeling functions; Each candidate pre-annotation result is format-converted to obtain a format-converted candidate pre-annotation result; the data structure of the format-converted candidate pre-annotation result is consistent with the data structure of the platform annotation object. Obtain at least one corrected pre-labeling result, wherein the corrected pre-labeling result is the result after the user corrects the candidate pre-labeling result after the format conversion; At least one of the corrected pre-annotation results is converted to a new format to obtain at least one converted corrected pre-annotation result; the data structure of the converted corrected pre-annotation result matches the annotation format of the target annotation engine. The platform pre-annotation result is obtained based on the corrected pre-annotation result after at least one format conversion.

[0137] In one alternative implementation, such as Figure 5 As shown, the data annotation device 400 further includes an incremental training module 406, which is used for: Based on the label type, error type, and invoked agent, the candidate pre-labeling results and the format-converted corrected pre-labeling results are clustered to determine multiple corrected datasets; If the amount of data in any of the modified datasets reaches a preset data amount threshold, an incremental training dataset is generated based on the modified dataset, and an incremental training prompt for the agent is issued. In response to receiving an incremental training instruction for the agent, the agent corresponding to the corrected dataset is trained based on the incremental training dataset.

[0138] In one alternative implementation, such as Figure 5 As shown, the data annotation device 400 further includes a bidirectional synchronization module 407, which is used for: In response to a change in the labeled object entity, the change information of the labeled object entity in the business platform is transmitted to the target labeling engine; In response to a change in the annotation results stored in the target annotation engine, the engine annotation result change information of the target annotation engine is transmitted to the business platform.

[0139] In one optional implementation, each entity in the platform data model is labeled with a data version number; such as... Figure 5 As shown, the data annotation device 400 further includes a collision detection module 408, which is used for: If a conflict is detected between the data version number and the update time of the target annotation engine, a conflict record is generated. The conflict record includes the platform data model of the business platform, the engine annotation result of the target annotation engine, and the difference field. The entity change information of the business platform is transmitted to the target annotation engine.

[0140] In one alternative implementation, such as Figure 5 As shown, the data annotation device 400 further includes a state management module 409, which is used for: Receive the user's review result for the target annotation results, the review result including review passed and review failed; If the audit result indicates that the audit is not approved, an issue item entity is created, and the issue item entity is marked with a status indicating that it is pending processing. In response to detecting that the user has modified an issue item entity whose status is marked as pending, the status of the issue item entity is switched to being processed. In response to detecting that the user has submitted a correction result for an issue item entity whose status is marked as "processing", the status of the issue item entity is switched to "pending review". In response to detecting the user's confirmation of the issue item entity marked as pending review, the status of the issue item entity is switched to closed; If any of the aforementioned problem items is marked as being processed and the status has not been switched to pending review after the preset processing time limit has expired, the status of the problem item will be switched to upgraded.

[0141] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0142] The data annotation apparatus provided in this disclosure establishes a business platform data model for data management, achieving unified and standardized management of the entire annotation business data chain. When annotation is required, it queries and calls the required target annotation engine based on a mapping table, effectively decoupling the business platform from the annotation engine and improving the business platform's adaptability and flexible expansion capabilities to multiple annotation engines. Thus, when it is necessary to replace or add annotation engines, only the engine adaptation layer and format converter need to be expanded, without modifying the main business process and main data structure of the business platform. This reduces the scope and cost of replacing annotation engines and accessing multiple annotation engines, and improves the stability of data annotation processing.

[0143] Corresponding to Figure 2 In addition to the data annotation method in this disclosure, this embodiment also provides a computer device 600, such as... Figure 6 The diagram shown is a structural schematic of a computer device 600 provided in an embodiment of this disclosure, comprising: The processor 610, memory 620, and bus 630 are included. The memory 620 is used to store execution instructions and includes main memory 621 and external memory 622. The main memory 621, also known as internal memory, is used to temporarily store the operation data in the processor 610 and the data exchanged with external memory 622 such as hard disk. The processor 610 exchanges data with external memory 622 through main memory 621.

[0144] In this embodiment, the memory 620 is specifically used to store application code that executes the solution of this application, and its execution is controlled by the processor 610. That is, when the computer device 600 is running, the processor 610 communicates with the memory 620 through the bus 630, or the processor 610 communicates with the memory 620 through other means, so that the processor 610 executes the application code stored in the memory 620, and then executes the steps of the data annotation method described in any of the foregoing embodiments.

[0145] The memory 620 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0146] Processor 610 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0147] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the computer device 600. In other embodiments of this application, the computer device 600 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0148] Furthermore, this disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the data annotation method described in the above-described method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0149] This disclosure also provides a computer program product, which stores a computer program. When the computer program is run by a processor, it executes the steps of the data annotation method provided in any of the above embodiments of this disclosure. For details, please refer to the above method embodiments, which will not be repeated here.

[0150] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium, which can be a volatile or non-volatile computer-readable storage medium. In another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0151] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices and apparatuses described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0152] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0153] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0154] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A data annotation method, characterized in that, The method is applied to a data annotation system, which includes a business platform and at least one annotation engine; the method includes: The platform receives data to be annotated through the business platform and constructs a platform data model based on the data to be annotated. The platform data model includes project entities, task entities, file entities, annotation object entities, annotation thread entities, issue item entities, and version snapshot entities. The business platform determines the target annotation engine that matches the data to be annotated based on a stored mapping table; the mapping table is used to indicate the mapping relationship between the business platform and the annotation engine; the mapping table includes platform project number, platform file number, platform annotation object number, engine type, engine project number, engine task number, and engine annotation number; The data to be labeled is pre-labeled through the business platform to obtain the platform pre-labeling results; The business platform generates a labeling configuration file that matches the target labeling engine based on historical labeling results of the same type of task as the data to be labeled and the platform's data model. The target annotation engine performs annotation processing on the annotation configuration file based on the platform's pre-annotation results to determine the target annotation result of the data to be annotated.

2. The method according to claim 1, characterized in that, The step of generating a labeling configuration file that matches the target labeling engine through the business platform based on historical labeling results of the same type of task as the data to be labeled and the platform's data model includes: Based on the platform's data model, the business platform determines the data modality of the data to be labeled and the labeling information of the project entities. Based on the data modality, determine the annotation control type that matches the data to be annotated; Based on the annotation label information, the annotation control metamodel corresponding to the annotation control type is configured to obtain a candidate metamodel; the annotation control metamodel includes type identifier, geometric dimension, attribute list, validation rules and rendering parameters; Based on the historical annotation results of the same type of task as the data to be annotated, the candidate meta-model is adjusted to obtain the target meta-model; Based on the target meta-model, an annotation configuration file matching the target annotation engine is generated.

3. The method according to claim 1, characterized in that, The step of pre-labeling the data to be labeled through the business platform to obtain the platform pre-labeling result includes: The business platform invokes at least one intelligent agent to perform pre-labeling processing on the data to be labeled, and obtains at least one candidate pre-labeling result; different intelligent agents have different labeling functions; Each candidate pre-annotation result is format-converted to obtain a format-converted candidate pre-annotation result; the data structure of the format-converted candidate pre-annotation result is consistent with the data structure of the platform annotation object. Obtain at least one corrected pre-labeling result, wherein the corrected pre-labeling result is the result after the user corrects the candidate pre-labeling result after the format conversion; At least one of the corrected pre-annotation results is converted to a new format to obtain at least one converted corrected pre-annotation result; the data structure of the converted corrected pre-annotation result matches the annotation format of the target annotation engine. The platform pre-annotation result is obtained based on the corrected pre-annotation result after at least one format conversion.

4. The method according to claim 3, characterized in that, The method further includes: Based on the label type, error type, and invoked agent, the candidate pre-labeling results and the format-converted corrected pre-labeling results are clustered to determine multiple corrected datasets; If the amount of data in any of the modified datasets reaches a preset data amount threshold, an incremental training dataset is generated based on the modified dataset, and an incremental training prompt for the agent is issued. In response to receiving an incremental training instruction for the agent, the agent corresponding to the corrected dataset is trained based on the incremental training dataset.

5. The method according to claim 1, characterized in that, The method further includes: In response to a change in the labeled object entity, the change information of the labeled object entity in the business platform is transmitted to the target labeling engine; In response to a change in the annotation results stored in the target annotation engine, the engine annotation result change information of the target annotation engine is transmitted to the business platform.

6. The method according to claim 1, characterized in that, The platform data model includes entities labeled with data version numbers; the method also includes: If a conflict is detected between the data version number and the update time of the target annotation engine, a conflict record is generated. The conflict record includes the platform data model of the business platform, the engine annotation result of the target annotation engine, and the difference field. The entity change information of the business platform is transmitted to the target annotation engine.

7. The method according to claim 1, characterized in that, The method further includes: Receive the user's review result for the target annotation results, the review result including review passed and review failed; If the audit result indicates that the audit is not approved, an issue item entity is created, and the issue item entity is marked with a status indicating that it is pending processing. In response to detecting that the user has modified an issue item entity whose status is marked as pending, the status of the issue item entity is switched to being processed. In response to detecting that the user has submitted a correction result for an issue item entity whose status is marked as "processing", the status of the issue item entity is switched to "pending review". In response to detecting the user's confirmation of the issue item entity marked as pending review, the status of the issue item entity is switched to closed; If any of the aforementioned problem items is marked as being processed and the status has not been switched to pending review after the preset processing time limit has expired, the status of the problem item will be switched to upgraded.

8. A data annotation device, characterized in that, The device is applied to a data annotation system, which includes a business platform and at least one annotation engine; the device includes: The model building module is used to receive data to be labeled through the business platform and build a platform data model based on the data to be labeled. The platform data model includes project entities, task entities, file entities, labeling object entities, annotation thread entities, issue item entities, and version snapshot entities. The engine determination module is used to determine the target annotation engine that matches the data to be annotated based on the stored mapping table of the business platform; the mapping table is used to indicate the mapping relationship between the business platform and the annotation engine; the mapping table includes platform project number, platform file number, platform annotation object number, engine type, engine project number, engine task number and engine annotation number; The platform pre-labeling module is used to pre-label the data to be labeled through the business platform to obtain the platform pre-labeling result; The annotation configuration module is used to generate an annotation configuration file that matches the target annotation engine based on the historical annotation results of the data to be annotated and the platform data model through the business platform. The result determination module is used to perform annotation processing on the annotation configuration file based on the platform's pre-annotation results using the target annotation engine, and determine the target annotation result of the data to be annotated.

9. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the data annotation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the data annotation method as described in any one of claims 1 to 7.

Images