Data labeling method and device, equipment and storage medium

Abstract

This disclosure provides a data annotation method, apparatus, device, and storage medium. The method includes: during data annotation by a user through an annotation interface, in response to detecting a retrieval trigger event, determining the retrieval data corresponding to the retrieval trigger event and the query information corresponding to the retrieval data; acquiring a primitive library; the primitive library stores multiple preset primitives, which are generated based on annotated annotation data; retrieving data from the primitive library based on the query information and the structural index of each preset primitive to obtain multiple candidate primitives; retrieving data from the multiple candidate primitives based on the retrieval data, the query information, and the content index of each candidate primitive to obtain multiple reference primitives; displaying multiple reference primitives in the annotation interface, and generating annotation reference information and a second annotation control structure corresponding to the retrieval data when the user selects a target primitive, thereby assisting the user's annotation process and effectively improving annotation efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more specifically, to a data annotation method, apparatus, device, and storage medium. Background Technology

[0002] With the continuous growth in demand for high-quality labeled data, various data labeling platforms are widely used. When labeling data, these platforms can adopt different labeling methods for different modalities of data and also have functions such as task distribution, result saving, and result review.

[0003] When annotating data, annotation is usually done only around the project, task, or document itself. For example, relying solely on the data presented by the project, task, or document itself and annotating based on human experience results in low annotation efficiency. Summary of the Invention

[0004] This disclosure provides at least one data annotation method, apparatus, device, and storage medium. It generates preset graphic elements based on annotated annotation data. During the user's data annotation process, the preset graphic elements can be retrieved by combining structural index and content index to assist the user's annotation process and improve annotation efficiency.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] This disclosure provides a data annotation method, the method comprising: During the process of a user annotating data through the annotation interface, in response to the detection of a retrieval trigger event, the retrieval data corresponding to the retrieval trigger event and the query information corresponding to the retrieval data are determined. Obtain a primitive library; the primitive library stores multiple preset primitives, which are generated based on labeled data. Each preset primitive is associated with a corresponding structure index and content index. The structure index includes organization number, permission scope, tag code, and tag path. The content index includes descriptive text, attribute text, tag name, media type, geographic location, and content vector. Based on the query information and the structural index of each preset graphic element, a search is performed in the graphic element library to obtain multiple candidate graphic elements; Based on the retrieval data, the query information, and the content index of each candidate graphic element, a retrieval is performed among multiple candidate graphic elements to obtain multiple reference graphic elements; The annotation interface displays the multiple reference graphic elements. When the user selects a target graphic element from the multiple reference graphic elements, the annotation reference information corresponding to the search data is generated based on the label name, label path, description text, and attribute text of the target graphic element. The first annotation control structure corresponding to the target graphic element is converted into a second annotation control structure that matches the search data. The annotation reference information and the second annotation control structure are used to assist the user in annotating the search data.

[0007] In one optional implementation, the step of searching among multiple candidate primitives based on the retrieval data, the query information, and the content index of each candidate primitive to obtain multiple reference primitives includes: Based on the query information and the content index of each candidate graphic element, a search is conducted among multiple candidate graphic elements according to the description text, the attribute text, the tag name, and the media type to obtain multiple first candidate graphic elements; Based on the query information and the content index of each candidate graphic element, a search is performed among multiple candidate graphic elements according to the geographical location to obtain multiple second candidate graphic elements; The retrieved data is content encoded to generate a query vector. Based on the query vector and the content index of each candidate graphic element, a search is performed among multiple candidate graphic elements according to the content vector to obtain multiple third candidate graphic elements. Based on multiple first candidate graphic elements, multiple second candidate graphic elements, and multiple third candidate graphic elements, multiple reference graphic elements are determined.

[0008] In one optional implementation, the geographic location includes latitude and longitude coordinates, address description text, and geocoding; Based on the query information and the content index of each candidate graphic element, a search is performed among multiple candidate graphic elements according to the geographical location to obtain multiple second candidate graphic elements, including: Based on the query information and the content index of each candidate graphic element, the positions of multiple candidate graphic elements are compared according to the latitude and longitude coordinates to obtain multiple candidate graphic elements with matching positions. Based on the query information and the content index of each candidate graphic element, a semantic comparison is performed on multiple candidate graphic elements that match the location according to the address description text to obtain multiple candidate graphic elements that match semantically. Based on the query information and the content index of each candidate graphic element, semantic comparison is performed on multiple candidate graphic elements that are semantically matched according to the geocoding to obtain multiple candidate graphic elements that are hierarchically matched. Based on multiple candidate primitives in hierarchical matching, multiple second alternative primitives are determined.

[0009] In one optional implementation, displaying the plurality of reference elements in the annotation interface includes: For each reference primitive, determine the tag matching degree, text similarity, primitive quality, time decay factor, citation popularity, spatial distance weight, and vector similarity corresponding to the reference primitive; The comprehensive score of the reference graphic element is determined based on the tag matching degree, text similarity, graphic element quality, time decay factor, citation popularity, spatial distance weight, and vector similarity corresponding to the reference graphic element. The reference elements are displayed in the annotation interface in descending order of the comprehensive score.

[0010] In one optional implementation, converting the first annotation control structure corresponding to the target primitive into a second annotation control structure that matches the retrieved data includes: When the target graphic element is an image graphic element, the coordinates indicated by the first annotation control structure corresponding to the target graphic element are converted into coordinates that match the size of the retrieved data. Based on the coordinates that match the size of the retrieved data, a second annotation control structure that matches the retrieved data is generated. When the target graphic element is a text graphic element, the start and end character offsets are determined based on the text matching results between the target graphic element and the search data. Based on the first annotation control structure corresponding to the target graphic element and the start and end character offsets, a second annotation control structure matching the search data is generated. When the target graphic element is an audio or video graphic element, based on the sampling rate, frame rate, and time axis offset, the start and end times indicated by the first annotation control structure corresponding to the target graphic element are converted into start and end times that match the time period corresponding to the search data. Based on the start and end times that match the time period corresponding to the search data, a second annotation control structure that matches the search data is generated. When the target primitive is a point cloud primitive, a second annotation control structure matching the retrieved data is generated based on the first annotation control structure corresponding to the target primitive and the coordinate system calibration parameters.

[0011] In one optional implementation, the content vector is generated through the following steps: Based on the media type of the annotation data corresponding to the preset graphic element, feature extraction is performed on the annotation data corresponding to the preset graphic element to obtain a modal feature vector; Based on the label codes and label paths corresponding to the preset graphic elements, a label path vector is generated; The content vector is generated based on the modal feature vector and the tag path vector.

[0012] In one optional implementation, the method further includes: If labeled data is detected, the review status of the labeled data is determined, and a preset primitive corresponding to the labeled data is generated. Determine the review pass rate, citation popularity, consistency score, correction feedback score, and time validity score corresponding to the preset graphic elements; The quality of the preset graphic elements is determined based on their review pass rate, citation popularity, consistency score, correction feedback score, and time effectiveness score. If the review status is "passed" and the quality of the graphic element is greater than the graphic element quality threshold, the preset graphic element is added to the graphic element library.

[0013] According to a second aspect of this application, embodiments of this disclosure also provide a data annotation apparatus, the apparatus comprising: The retrieval triggering module is used to determine the retrieval data corresponding to the retrieval triggering event and the query information corresponding to the retrieval data in response to the detection of a retrieval triggering event during the process of a user annotating data through the annotation interface. The primitive library acquisition module is used to acquire a primitive library; the primitive library stores multiple preset primitives, which are generated based on labeled data. Each preset primitive is associated with a corresponding structure index and content index. The structure index includes organization number, permission scope, tag code, and tag path. The content index includes descriptive text, attribute text, tag name, media type, geographic location, and content vector. The candidate element retrieval module is used to search the element library based on the query information and the structural index of each preset element to obtain multiple candidate elements. The reference element retrieval module is used to retrieve multiple candidate elements based on the retrieval data, the query information, and the content index of each candidate element, to obtain multiple reference elements. The primitive display module is used to display the multiple reference primitives in the annotation interface, and when it detects that the user selects a target primitive from the multiple reference primitives, it generates annotation reference information corresponding to the search data based on the label name, label path, description text and attribute text of the target primitive, and converts the first annotation control structure corresponding to the target primitive into a second annotation control structure that matches the search data; the annotation reference information and the second annotation control structure are used to assist the user in annotating the search data.

[0014] According to a third aspect of this application, a computer device is provided, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the data annotation method described in the first aspect are performed.

[0015] According to a fourth aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, the computer program being executed by a computer device to perform the steps of the data annotation method described in the first aspect.

[0016] The data annotation method, apparatus, device, and storage medium provided in this disclosure pre-converts annotated data into preset primitives carrying structural and content indexes. During the user's data annotation process, when a retrieval trigger event is detected, a mixed retrieval can be performed based on the retrieval data and query information, according to the structural and content indexes. The retrieved reference primitives are displayed, and they can be reused according to the target primitives selected by the user to generate annotation reference information and annotation control structures corresponding to the retrieval data. This reduces the frequency of users manually adjusting the annotation control structure, assists users in annotation processing, and effectively improves annotation efficiency. Furthermore, by displaying reference primitives, it helps different annotators to have a consistent understanding and execution of the same tagging system, reduces annotation deviations caused by subjective judgment, ensures the consistency of organizational-level tagging, and improves annotation quality.

[0017] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure.

[0018] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0020] Figure 1 A flowchart of a data annotation method provided by an embodiment of this disclosure is shown; Figure 2 A schematic diagram of a data annotation system provided in an embodiment of this disclosure is shown; Figure 3 This illustration shows a data annotation process provided by an embodiment of the present disclosure; Figure 4 One of the schematic diagrams of a data annotation apparatus provided in an embodiment of this disclosure is shown; Figure 5 A second schematic diagram of a data annotation device provided in an embodiment of this disclosure is shown; Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0023] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0024] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0025] All user-related data involved in the technical solutions of this application are data authorized by the user. It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, the user should be informed of the type, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and the user's authorization should be obtained.

[0026] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the computer device, application program, server, or storage medium, or other software or hardware performing the operations of this disclosed technical solution.

[0027] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the computer device.

[0028] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0029] Research has revealed that when using data annotation platforms, annotation management typically revolves around individual projects, tasks, or files. Annotators primarily rely on the raw data displayed on the current page, selecting appropriate annotation controls from a list based on their experience. Even though the data platform has accumulated a large number of historical annotation results, these results are only used for exporting datasets, training models, or generating statistical reports; they are not transformed into searchable and reusable reference annotation resources during the annotation process.

[0030] Taking data annotation platforms such as Label Studio, ComputerVision Annotation Tool (CVAT), and Labelbox as examples, their core capabilities are mostly concentrated on annotation controls, task distribution, result saving, and basic review, lacking the ability to retrieve and reuse historical annotation results. When processing new annotation tasks, it is impossible to query historical annotation results related to the current annotation task for reference.

[0031] Because annotation is based on human experience, different annotators within the same organization may have varying understandings and implementations of the same tagging system. The lack of searchable historical annotation references leads to inconsistent application of annotation standards for the same tags by different annotators within the same organization. This is particularly problematic for complex data, long-tail tags, or industry-specific scenarios, where annotation quality heavily relies on the personal experience of a few senior annotators, making it difficult to guarantee organization-wide annotation quality.

[0032] Based on the above research, this disclosure provides a data annotation method that pre-converts annotated data into preset primitives carrying structural and content indexes. During the data annotation process, users can perform mixed retrieval of preset primitives to assist in the annotation process, effectively improve annotation efficiency, and help different annotators to have a consistent understanding and execution of the same tag system, reduce annotation deviations caused by subjective judgment, ensure the consistency of organizational-level tag usage, and improve annotation quality.

[0033] To facilitate understanding of this embodiment, a data annotation method disclosed in this disclosure will first be described in detail. The data annotation method provided in this disclosure generally executes a computer device with certain computing capabilities. This computer device can be a server, which can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms. In some possible implementations, the computer device can also be a terminal device, such as user equipment (UE), mobile device, user terminal, terminal, handheld device, computing device, wearable device, etc. In other implementations, this data annotation method can be applied to an implementation environment composed of a terminal device and a server. Furthermore, this data annotation method can also be implemented by a processor calling computer-readable instructions stored in memory.

[0034] The following description, in conjunction with the accompanying drawings, illustrates a data annotation method provided by an embodiment of this disclosure.

[0035] See Figure 1 The diagram shown is a flowchart of a data annotation method provided in an embodiment of this disclosure. Figure 1 As shown in the figure, the data annotation method provided in this embodiment includes steps S101 to S105, wherein: S101: During the process of a user annotating data through the annotation interface, in response to the detection of a retrieval trigger event, the retrieval data corresponding to the retrieval trigger event and the query information corresponding to the retrieval data are determined.

[0036] Here, the user can annotate data through the data annotation system. The user can be an annotator. See also... Figure 2 The diagram shown is a schematic representation of a data annotation system provided in an embodiment of this disclosure. Figure 2 As shown, the data annotation system 200 includes an annotation workbench 210. In multimodal annotation scenarios, the annotation workbench 210 can provide various annotation controls such as image rectangles / polygons / keypoints, text entities / document classifications, audio / video time segments, point cloud target boxes / segments, and has functions such as task distribution, result saving, basic review, and version management.

[0037] The data annotation system 200 may further include an annotation engine. Data annotation can be performed through the annotation engine. The data annotation system 200 can realize bidirectional synchronization of results between the annotation workbench 210 and the annotation engine, parent-child tag system, review and annotation, issue item construction, version tracking, audio and video timeline annotation, breakpoint resume, and artificial intelligence (AI) pre-annotation capabilities.

[0038] The annotation workbench 210 can display an annotation interface. An annotation reference interface can be displayed within the annotation interface. When multiple reference graphic elements are displayed in the annotation interface, these reference graphic elements are also displayed in the annotation reference interface, providing users with a search entry point and search results display. For example, the annotation reference interface can take the form of a floating window, pop-up window, sidebar, pull-up panel, sequence browsing bar, etc.

[0039] Here, the retrieval trigger event can include retrieval trigger events in active query mode and retrieval trigger events in context-aware active recommendation mode. Specifically, the retrieval trigger event in active query mode includes the user actively opening the annotation reference interface, for example, by using a shortcut key or clicking a button. When the retrieval trigger event is the user actively opening the annotation reference interface, the corresponding retrieval data consists of the tags, keywords, attributes, media types, etc., selected by the user when the user actively opens the annotation reference interface. The media type represents the data modality, such as image, text, audio, video, point cloud, etc.

[0040] In the context-aware proactive recommendation mode, retrieval trigger events include user opening a file through the annotation workbench, tag selection, image region selection, text segment selection, audio time segment selection, video time segment selection, and point cloud object selection. When the retrieval trigger event is a user opening a file through the annotation workbench, the corresponding retrieval data includes the media type of the opened file, the tags of the project it belongs to, and the domain of the project. When the retrieval trigger event is tag selection, the corresponding retrieval data is the selected tag. When the retrieval trigger event is image region selection, the corresponding retrieval data is the visual features of the selected image region; optionally, the visual features can be extracted from the selected image region using a visual encoder. When the retrieval trigger event is text segment selection, the corresponding retrieval data is the selected text segment. When the retrieval trigger event is audio time segment selection, the corresponding retrieval data is the selected audio time segment. When the retrieval trigger event is video time segment selection, the corresponding retrieval data is the selected video time segment. When the retrieval trigger event is point cloud object selection, the corresponding retrieval data is the selected point cloud object.

[0041] The query information corresponding to the retrieved data includes the organization number, user permissions, project number (Identity, ID), media type, tag code, tag path, selected area content, attribute constraints, geographical location, and file characteristics.

[0042] S102: Obtain the primitive library; the primitive library stores multiple preset primitives, which are generated based on labeled data. Each preset primitive is associated with a corresponding structure index and content index. The structure index includes organization number, permission scope, tag code, and tag path. The content index includes descriptive text, attribute text, tag name, media type, geographic location, and content vector.

[0043] In this step, each labeled data can be converted into a searchable glyph object in advance to generate the preset glyph, which is an atomic unit of organizational-level labeled knowledge.

[0044] The preset primitives include at least the following fields: primitive ID, organization number, source project ID, source task ID, source file ID, media type, tag name, tag code, tag path, parent tag code, target type, modal range, attribute text, description text, review status, permission scope, version number, quality score, creation time, update time, source annotation object ID, thumbnail, and clip preview address. The modal range is normalized coordinates or a point sequence in image modality, start and end character offsets in text modality, start and end timestamps in audio or video modality, and a 3D bounding box or point set index in point cloud modality. The permission scope includes organization-level public, project-level shared, and personal private permissions.

[0045] See again Figure 2 The data annotation system 200 also includes a primitive library 220. The primitive library 220 stores multiple preset primitives. Optionally, the primitive library itself, as a special annotation type, has independent annotation, review, rejection, approval, version tracking, and issue handling processes. The primitive library administrator can perform quality reviews of the preset primitives already in the library. If problems such as incorrect labels or inaccurate boundaries are found, a rejection operation can be performed and associated issue items generated. Each modification to a preset primitive already in the library generates a new version number, and older versions can be traced and rolled back.

[0046] Here, a unified retrieval index can be built within the organization for the multiple preset graphic elements, thereby supporting multimodal and multi-dimensional hybrid retrieval. Optionally, index construction can be performed on the preset graphic elements when they are added to the database or when their versions are updated.

[0047] Optionally, before building an index for the preset graphic elements, the organization number, permission scope, review status, and version number of the preset graphic elements can be verified first. After the verification is passed, the index for the preset graphic elements can be built.

[0048] Specifically, a structural index and a content index can be constructed for the preset graphic elements. The structural index includes organization number, permission scope, tag code, and tag path. The content index includes a full-text index and a vector index. The full-text index includes descriptive text, attribute text, tag name, media type, and geographic location. The vector index includes content vectors.

[0049] Optionally, the structure index may also include project number, review status, version number, and creation time. The full-text index may also include address text.

[0050] Here, by writing the organization number into the structure index, it is ensured that the search results come from the corresponding organization. By writing the permission scope into the structure index, it is ensured that the search results satisfy the user's permissions.

[0051] Optionally, all preset elements added to the element library have a "passed" review status and a valid version number, preventing rejected, frozen, or deleted elements from being retrieved and ensuring the quality of stored data in the element library. The element library administrator can view rejected, frozen, or deleted elements through other methods.

[0052] In traditional data annotation methods, the annotated data is not indexed uniformly. Users cannot retrieve the annotated data based on the index during the annotation process, nor can they use the retrieved primitives as reference objects for the current annotation. There is a lack of organizational-level and cross-task annotation knowledge retrieval capabilities.

[0053] This disclosure converts labeled data into searchable preset graphic elements, enabling users to directly call existing graphic elements and their related information when processing similar or identical labeled data, reducing redundant labeling, and allowing complex graphic elements, long-tail labels, and industry experience to be accumulated and reused, thereby improving labeling efficiency.

[0054] In some possible implementations, the content vector is generated through the following steps: Based on the media type of the annotation data corresponding to the preset graphic element, feature extraction is performed on the annotation data corresponding to the preset graphic element to obtain a modal feature vector; Based on the label codes and label paths corresponding to the preset graphic elements, a label path vector is generated; The content vector is generated based on the modal feature vector and the tag path vector.

[0055] In the above steps, the content vectors of each preset primitive can be generated by a trained primitive encoder, thereby mapping preset primitives of different modalities to a unified vector space. The primitive encoder includes a modal feature encoding module, a label path encoding module, and a vector fusion module. The modal feature encoding module calls the encoder corresponding to the media type of the labeled data corresponding to the preset primitive according to the media type, and extracts features from the labeled data corresponding to the preset primitive to obtain a modal feature vector. Specifically, for image primitives, the encoder input is the cropped target area or mask area, and the output is a visual feature vector; for text primitives, the encoder input is the entity fragment and its context window, and the output is a semantic feature vector; for audio primitives, the encoder input is the acoustic features within the start and end time period, and the output is an audio feature vector; for video primitives, the encoder input is the keyframe sequence and time period, and the output is a spatiotemporal feature vector; for point cloud primitives, the encoder input is a 3D point set or point cloud within a bounding box, and the output is a spatial structure feature vector.

[0056] The label path encoding module maps the label codes of each level, from the root label to the leaf label, corresponding to the preset primitive to the corresponding label embedding vectors. Based on the depth of each level label code in the label path, a corresponding weight coefficient is assigned to each level, wherein the weight coefficient corresponding to the deeper label is greater than the weight coefficient corresponding to the shallower label. The label embedding vectors are weighted and pooled according to the corresponding weight coefficients to obtain the label path vector that integrates the structural information of each level.

[0057] The vector fusion module normalizes the modal feature vector and the tag path vector, respectively, and then concatenates or weights the normalized modal feature vector and the normalized tag path vector to obtain the content vector.

[0058] During training, the primitive encoder uses similar primitives with the same label path that have passed review as positive samples and primitives with different label paths or different semantic categories as negative samples. It employs contrastive learning loss to narrow the distance between similar primitive vectors and widen the distance between dissimilar primitive vectors.

[0059] In this way, through the processing of the primitive encoder, the content vectors of multiple preset primitives that are cross-modal and semantically similar are located close to each other in a unified vector space. For example, the content vectors corresponding to the rectangular box of "car" in the image, the entity segment of "car" in the text, and the audio segment of "car" in the audio are located close to each other in the unified vector space.

[0060] When searching in any modality, semantically similar primitives from all modalities can be retrieved, enabling cross-modal reference reuse. For example, when the retrieved data is "car" in an image, the text primitive "car" can be retrieved.

[0061] In this way, the content vector possesses both the modal semantic information of the labeled data itself and its structured position information in the tag system. Thus, in the subsequent hybrid retrieval process, the content vector can comprehensively measure the matching degree between the preset primitives and the retrieved data from two dimensions: semantic similarity and tag hierarchical correlation. This improves the relevance of the retrieval results to the retrieved data at both the content and tag levels, helps to improve the recall accuracy of the reference primitives, and provides annotators with more accurate annotation references, thereby improving annotation efficiency and quality.

[0062] S103: Based on the query information and the structural index of each preset graphic element, a search is performed in the graphic element library to obtain multiple candidate graphic elements.

[0063] See again Figure 2 The data annotation system 200 also includes a retrieval engine 230. Retrieval requests can be submitted to the retrieval engine 230, for example, asynchronously. This asynchronous submission of retrieval requests eliminates the need for the main annotation process to wait for retrieval results, avoiding interface lag caused by network or processing delays in synchronous requests and ensuring the continuity of annotation operations. Simultaneously, it decouples the retrieval task from the main annotation task, allowing the retrieval engine to process retrieval requests independently of the annotation engine, avoiding concentrated consumption of system resources and contributing to improved system throughput and operational stability. The retrieval request carries retrieval data corresponding to the retrieval trigger event and query information corresponding to the retrieval data. The retrieval engine 230, based on the query information and the structural indexes of each preset graphic element, searches the graphic element library to obtain multiple candidate graphic elements.

[0064] In this step, based on the query information and the structural index of each preset graphic element, a mixed search can be performed in the graphic element library according to the organization number, permission scope, tag code, and tag path, and multiple candidate graphic elements can be determined through multi-way recall.

[0065] Specifically, based on the query information and the structural index of each preset graphic element, a search can be performed in the graphic element library according to the organization number and the permission scope to obtain multiple first pre-reserved graphic elements; the first pre-reserved graphic elements are preset graphic elements that belong to this organization and whose permission scope meets the user's permissions.

[0066] Based on the query information and the structural index of each preset graphic element, a search is performed in the graphic element library according to the tag code to obtain multiple second preparatory graphic elements; the second preparatory graphic elements are preset graphic elements that match the tag codes included in the query information.

[0067] Based on the query information and the structural indexes of each preset graphic element, a search is performed in the graphic element library according to the tag path to obtain multiple third-prepared graphic elements; the third-prepared graphic elements are subclass graphic elements determined based on the tag paths included in the query information. Here, all subclass graphic elements under a parent tag can be searched (path prefix matching), or a specific sub-level tag can be searched (exact match). Optionally, whether to search for a certain level of subclass graphic elements can be determined according to user instructions.

[0068] Based on multiple first-stage, second-stage, and third-stage pre-selected graphic elements, multiple candidate graphic elements are determined. Specifically, these multiple first-stage, second-stage, and third-stage pre-selected graphic elements can be merged and deduplicated, for example, by deduplicating them according to graphic element ID and version number, to determine multiple candidate graphic elements.

[0069] S104: Based on the retrieval data, the query information, and the content index of each candidate graphic element, a retrieval is performed among multiple candidate graphic elements to obtain multiple reference graphic elements.

[0070] In this step, the retrieval engine can be used to search among multiple candidate graphic elements based on the retrieval data, the query information, and the content index of each candidate graphic element to obtain multiple reference graphic elements. Specifically, based on the retrieval data, the query information, and the content index of each candidate graphic element, a mixed search can be performed among multiple candidate graphic elements according to descriptive text, attribute text, tag name, media type, geographic location, and content vector, and multiple reference graphic elements can be determined through multi-path recall.

[0071] In some possible implementations, the retrieval of multiple candidate primitives based on the retrieval data, the query information, and the content index of each candidate primitive to obtain multiple reference primitives includes: Based on the query information and the content index of each candidate graphic element, a search is conducted among multiple candidate graphic elements according to the description text, the attribute text, the tag name, and the media type to obtain multiple first candidate graphic elements; Based on the query information and the content index of each candidate graphic element, a search is performed among multiple candidate graphic elements according to the geographical location to obtain multiple second candidate graphic elements; The retrieved data is content encoded to generate a query vector. Based on the query vector and the content index of each candidate graphic element, a search is performed among multiple candidate graphic elements according to the content vector to obtain multiple third candidate graphic elements. Based on multiple first candidate graphic elements, multiple second candidate graphic elements, and multiple third candidate graphic elements, multiple reference graphic elements are determined.

[0072] In the above steps, based on the query information and the content index of each candidate graphic element, a search can be performed among multiple candidate graphic elements according to the descriptive text, the attribute text, the tag name, and the media type, respectively, to obtain multiple first candidate graphic elements. When searching according to the descriptive text, the attribute text, and the tag name, keyword matching can be performed. When searching according to the media type, filtering can be performed according to image, text, audio, video, and point cloud.

[0073] Optionally, in addition to searching among multiple candidate graphic elements according to the description text, the attribute text, the label name, and the media type respectively, the attribute text can also be filtered by equal value or range according to user-defined attribute fields, such as color=white and occlusion=no, to obtain multiple first candidate graphic elements.

[0074] Based on the query information and the content index of each candidate graphic element, a search is conducted among multiple candidate graphic elements according to the geographical location to obtain multiple second candidate graphic elements.

[0075] The retrieved data is content-encoded to generate a query vector. Here, the primitive encoder can be used to content-encode the retrieved data to generate the query vector. Based on the query vector and the content index of each candidate primitive, a search is performed among multiple candidate primitives according to the content vector to obtain multiple third candidate primitives. The vector similarity between the content vector and the query vector of each candidate primitive can be determined; for example, the cosine similarity between the content vector and the query vector can be used as the vector similarity. A first preset number of candidate primitives with the highest vector similarity are determined as the third candidate primitives. The specific number of these first preset numbers can be set according to actual data retrieval needs and is not specifically limited here.

[0076] Based on multiple first-option, second-option, and third-option graphic elements, multiple reference graphic elements are determined. Specifically, the multiple first-option, second-option, and third-option graphic elements can be merged and deduplicated, for example, by deduplicating according to graphic element ID and version number, to determine multiple reference graphic elements.

[0077] In this way, retrieval can be performed from multiple dimensions, avoiding the failure to recall relevant primitives due to insufficient feature representation in a single retrieval dimension, thereby expanding the retrieval coverage and improving the recall rate and diversity of primitives.

[0078] In some possible implementations, a downgraded search is performed if the number of identified reference elements is less than a first threshold or if no reference elements are returned within a preset time. Specifically, if the number of initially identified reference elements is less than the first threshold or if no reference elements are returned within a preset time, a first-level downgraded search can be performed, replacing the current tag with its parent tag and then performing the search again. If the number of second identified reference elements is less than the first threshold or if no reference elements are returned within a preset time, a second-level downgraded search can be performed, canceling the search based on the description text and the attribute text. If the number of third identified reference elements is less than the first threshold or if no reference elements are returned within a preset time, a third-level downgraded search can be performed, increasing the value of the first preset number to expand the recall scope of the content vector-based similarity search. The first threshold can be set according to actual data retrieval needs and is not specifically limited here.

[0079] Optionally, during the process of performing a downgraded search, a downgraded prompt message can be displayed in the annotation interface. Specifically, an annotation reference interface can be displayed in the annotation interface, and the downgraded prompt message can be displayed in the annotation reference interface. The downgraded prompt message may be, for example, "Not enough similar graphic elements have been retrieved, and reference graphic elements of the superior category have been automatically displayed."

[0080] In traditional data annotation methods, the spatial dimension of geographic location is not effectively utilized in the annotation process. In annotation scenarios involving geographic information, such as remote sensing image annotation, street view image annotation, and geographic entity annotation, the annotated data usually has clear geographic location information. However, data annotation platforms lack the ability to retrieve historical annotated data based on geographic location, and the spatial distribution information of annotations fails to serve as a reference for retrieval.

[0081] The embodiments disclosed herein can perform searches based on geographical location, thus expanding the spatial dimension search capabilities.

[0082] In some possible implementations, the geographic location includes latitude and longitude coordinates, address description text, and geocoding. The latitude and longitude coordinates include longitude and latitude coordinates. The geocoding includes administrative division codes, with administrative division levels such as province, city, and street. Optionally, the geographic location also includes location source and coordinate precision. The location source may be, for example, Global Positioning System (GPS), manual annotation, or reverse geocoding. The coordinate precision may be, for example, meters or kilometers.

[0083] Based on the query information and the content index of each candidate graphic element, a search is performed among multiple candidate graphic elements according to the geographical location to obtain multiple second candidate graphic elements, including: Based on the query information and the content index of each candidate graphic element, the positions of multiple candidate graphic elements are compared according to the latitude and longitude coordinates to obtain multiple candidate graphic elements with matching positions. Based on the query information and the content index of each candidate graphic element, a semantic comparison is performed on multiple candidate graphic elements that match the location according to the address description text to obtain multiple candidate graphic elements that match semantically. Based on the query information and the content index of each candidate graphic element, semantic comparison is performed on multiple candidate graphic elements that are semantically matched according to the geocoding to obtain multiple candidate graphic elements that are hierarchically matched. Based on multiple candidate primitives in hierarchical matching, multiple second alternative primitives are determined.

[0084] In the above steps, multiple candidate map elements with matching locations can be obtained by matching latitude and longitude coordinates. Then, based on the address description text, semantic matching is performed on the semantic tags (such as business districts, residential areas, etc.) automatically labeled by the map annotation point (POI) data to obtain multiple candidate map elements with matching semantics. Next, multiple candidate map elements with matching levels are obtained by geocoding.

[0085] Optionally, candidate primitives with hierarchical matching can be determined as the second candidate primitives. Alternatively, candidate primitives with hierarchical matching located within a user-defined spatial range can be determined as the second candidate primitives. For example, a user can draw a polygonal region and determine the candidate primitives with hierarchical matching within that region as the second candidate primitives. For instance, a user can instruct the retrieval of primitives within a radius of N meters of the current location.

[0086] In this way, multi-level dimensional retrieval under geographic location can filter irrelevant primitives at different spatial granularities, avoiding false recall or missed recall caused by a single spatial matching method, improving the accuracy of geospatial dimension retrieval results, providing reference primitives that are more relevant to the current search data in spatial location for annotation scenarios involving geographic information, effectively enriching the auxiliary reference for geospatial dimension annotation, and thus enhancing the decision-making basis of annotators in the spatial dimension.

[0087] In some possible implementations, the annotation interface can display a regional map corresponding to a user-defined spatial range, showing the annotation distribution, aggregation trends, and heatmap areas. Specifically, multi-dimensional filtering can be performed based on organization, project, tag, parent-child tags, element type, and review status, displaying the corresponding annotation distribution on the regional map. When the number of reference elements in the regional map exceeds a second quantity threshold, aggregators can be obtained by tag type to obtain quantity markers, which are then displayed on the regional map. The quantity markers can change their display state based on user triggers, such as zooming and expanding. The second quantity threshold can be set according to actual data retrieval needs and is not specifically limited here. A heatmap can be generated based on the spatial density of reference elements in the regional map to visually display densely distributed annotation areas.

[0088] Optionally, in response to triggering a reference element marker in the area map, details of the reference element can be viewed and the user can jump to its corresponding annotation task context.

[0089] In some possible implementations, the spatial distribution state of each label can be determined based on the distribution of preset primitives in the primitive library. When the deviation between the geographical location of the retrieved data and the spatial distribution state of its corresponding label type exceeds a preset threshold, the user can be prompted for confirmation. The preset threshold can be set according to the actual data retrieval needs and is not specifically limited here. Thus, spatial distribution anomaly detection helps improve labeling quality.

[0090] S105: Display the plurality of reference graphic elements in the annotation interface, and when the user selects a target graphic element from the plurality of reference graphic elements, generate annotation reference information corresponding to the search data based on the label name, label path, description text and attribute text of the target graphic element, and convert the first annotation control structure corresponding to the target graphic element into a second annotation control structure that matches the search data; the annotation reference information and the second annotation control structure are used to assist the user in annotating the search data.

[0091] In this step, a reference annotation interface can be displayed within the annotation interface, showcasing the multiple reference graphic elements. Specifically, the complete reference graphic element can be displayed, or a thumbnail or preview fragment can be shown. This allows users to view the complete reference graphic element when detailed comparison is needed, and to save interface display space and reduce the frequency of scrolling or switching operations when quickly browsing multiple reference graphic elements is required, thereby improving the efficiency of viewing and selecting reference graphic elements.

[0092] Optionally, in addition to displaying the reference graphic elements, the label path (e.g., transportation - vehicle - sedan), source project name, source task name (which can be used to jump to view the original task context), review status (e.g., approved, rejected), explanatory text, citation popularity, and graphic element quality of the reference graphic elements can also be displayed.

[0093] This embodiment of the disclosure displays multiple reference elements in the annotation interface, enabling annotators to directly refer to historical annotation results. This avoids the need to repeatedly consult documents or senior personnel due to unfamiliarity with annotation specifications, thereby reducing reliance on manual guidance during the annotation process. It also helps annotators learn annotation standards independently through readily available reference examples, shortens the learning cycle, reduces annotation training costs, and improves annotation efficiency.

[0094] In some possible implementations, displaying the plurality of reference elements in the annotation interface includes: For each reference primitive, determine the tag matching degree, text similarity, primitive quality, time decay factor, citation popularity, spatial distance weight, and vector similarity corresponding to the reference primitive; The comprehensive score of the reference graphic element is determined based on the tag matching degree, text similarity, graphic element quality, time decay factor, citation popularity, spatial distance weight, and vector similarity corresponding to the reference graphic element. The reference elements are displayed in the annotation interface in descending order of the comprehensive score.

[0095] Here, the label matching degree corresponding to the reference element can be determined based on the matching of the retrieved data with the label of the reference element. If the label corresponding to the retrieved data matches the label corresponding to the reference element, the label matching degree is 1. If the parent label corresponding to the retrieved data matches the label corresponding to the reference element, the label matching degree is 0.7. If any child label at any level corresponding to the retrieved data matches the label corresponding to the reference element, the depth ratio of the matching level to the label path is determined as the label matching degree.

[0096] The text similarity can be determined using BM25 or vector text similarity.

[0097] The time decay factor can be determined by exponentially decaying the time difference between the most recent approval time of the reference element and the current retrieval time. Specifically, the time decay factor can be determined by the following formula (1): (1) in, Indicates the time decay factor. Indicates the attenuation rate. This indicates the time difference between the most recent approval time of the reference element and the current retrieval time.

[0098] In this way, the greater the time difference between the most recent approval time of the reference graphic element and the current retrieval time, the earlier the approval time of the preset graphic element is, and the smaller the corresponding time decay factor is. This gives the preset graphic elements that have recently been approved a higher weight in the comprehensive score, so that the timeliness of the reference graphic element can be quantitatively reflected.

[0099] The quantile values ​​of the number of references of all preset elements within the organization where the reference element is located can be statistically analyzed. The number of references of the reference element can be mapped to the normalized value corresponding to its quantile position. For example, after sorting the number of references from low to high, the references can be converted into normalized values ​​in the range of 0 to 1 according to their quantile positions. The normalized value is determined as the reference popularity. This allows reference elements with different reference frequencies to quantify their degree of reuse within the organization, reflecting the reference value of the preset elements in the historical annotation process within the organization.

[0100] If the search data does not carry geographic location information, the spatial distance weight can be set to 1. If the search data carries geographic location information, the spatial distance weight is determined by a preset distance decay function based on the spatial distance between the latitude and longitude coordinates corresponding to the reference image element and the geographic location in the query information. The smaller the spatial distance, the larger the spatial distance weight; however, the spatial distance weight decreases as the spatial distance increases but does not drop to zero, ensuring that reference images located far away but with matching labels still have a chance to be viewed by the user. The preset distance decay function can be, for example, a Gaussian decay function, a linear decay function, or a step decay function, to support distance decay strategy configurations for different application scenarios.

[0101] The vector cosine similarity between the retrieved data and the reference primitive can be determined to obtain the vector similarity.

[0102] The review pass rate, citation popularity, consistency score, correction feedback score, and time validity score corresponding to the reference graphic element can be determined. Based on the review pass rate, citation popularity, consistency score, correction feedback score, and time validity score corresponding to the reference graphic element, the graphic element quality of the reference graphic element can be determined.

[0103] Specifically, the approval rate of a reference element can be determined by the ratio of the number of times it has passed review to the total number of reviews. The consistency score of a reference element can be determined by the degree of deviation of its geometric dimensions, attribute values, or text length from the statistical center of the same tag path. The correction feedback score of a reference element can be determined by the number of historical corrections; the more historical corrections, the lower the correction feedback score. The time validity score can be determined by exponentially decaying the most recent approval time of the reference element.

[0104] Here, the weights corresponding to the review pass rate, citation popularity, consistency score, correction feedback score, and time validity score can be determined. These weights are then weighted and summed to determine the quality of the reference graphic element. The weights for each of these factors can be set based on search quality and are not specifically limited here. Optionally, version numbers of the weights can be saved to ensure the ranking process for the same search request is traceable.

[0105] Specifically, the quality of the primitive can be determined by the following formula (2): (2) in, Indicates the quality of the primitive. Indicates the approval rate. This indicates the weight corresponding to the approval rate. Indicates the popularity of citations. This indicates the weight corresponding to the popularity of citations. Indicates consistency score, This indicates the weight corresponding to the consistency score. This indicates a correction to the feedback rating. This indicates the weighting corresponding to the adjusted feedback score. Indicates time validity score, This indicates the weight corresponding to the time effectiveness score.

[0106] The weights corresponding to tag matching degree, text similarity, primitive quality, time decay factor, citation popularity, spatial distance weight, and vector similarity can be determined. These weights are then weighted and summed to determine the comprehensive score of the reference primitive. The weights for tag matching degree, text similarity, primitive quality, time decay factor, citation popularity, spatial distance weight, and vector similarity can be set according to search quality and are not specifically limited here. Optionally, version numbers of the weights can be saved to ensure the ranking process for the same search request is traceable.

[0107] To reduce discrepancies between different data sets, label matching degree, text similarity, primitive quality, time decay factor, citation popularity, spatial distance weight, and vector similarity can be normalized. This ensures that the normalized label matching degree, text similarity, primitive quality, time decay factor, citation popularity, spatial distance weight, and vector similarity are all between 0 and 1. When determining the comprehensive score of the reference primitive by weighted summation of these factors according to their respective weights, specifically, the comprehensive score can be determined by weighted summation of the normalized label matching degree, text similarity, primitive quality, time decay factor, citation popularity, spatial distance weight, and vector similarity.

[0108] Specifically, the comprehensive score can be determined by the following formula (3): (3) in, This indicates the overall score. Indicates the degree of tag matching. This represents the weight corresponding to the tag matching degree. Indicates text similarity. The weights representing the text similarity are... Indicates the quality of the primitive. The weights represent the quality of the primitives. Indicates the time decay factor. This indicates the weight corresponding to the time decay factor. Indicates the popularity of citations. This indicates the weight corresponding to the popularity of citations. Indicates spatial distance weights. This represents the weight corresponding to the spatial distance weight. Represents vector similarity, The weights represent the vector similarity.

[0109] In this way, by determining the tag matching degree, text similarity, icon quality, time decay factor, citation popularity, spatial distance weight, and vector similarity for each reference icon, and determining the comprehensive score of the reference icon based on the above multiple dimensions of factors, and then displaying multiple reference icons in the annotation interface in descending order of comprehensive score, the reference icons with higher comprehensive relevance to the current search data in multiple dimensions such as tag standardization, text semantics, icon quality, timeliness, citation frequency, spatial location, and vector semantics can be viewed by users first. This reduces the operation of browsing and filtering among many reference icons, helps users quickly locate reference icons with higher matching degree, and improves the selection efficiency of target icons.

[0110] In some possible implementations, maximum marginal relevance (MMR) can be used to perform diversity rearrangement on multiple reference primitives, resulting in a selection of reference primitives; these selected reference primitives are then displayed in the annotation interface. Specifically, the selected reference primitives can be displayed in the annotation interface in descending order of their comprehensive scores.

[0111] Specifically, a candidate primitive set can be formed by a second preset number of reference primitives with the highest comprehensive scores; for each reference primitive in the candidate primitive set, a diversity score is determined based on the comprehensive score of the reference primitive and the maximum vector similarity between the reference primitive and each reference primitive in the result primitive set; the reference primitive with the highest diversity score is moved from the candidate primitive set to the result primitive set; the steps of determining the diversity score of each reference primitive in the candidate primitive set based on the comprehensive score of the reference primitive and the maximum vector similarity between the reference primitive and each reference primitive in the result primitive set are repeated until the number of reference primitives in the result primitive set reaches a third preset number.

[0112] Here, for the first iteration, the result primitive set is empty, and at this time, the maximum vector similarity between the reference primitive and each reference primitive in the result primitive set can be determined as 0.

[0113] The second preset quantity is greater than the third preset quantity. The specific values ​​of the second and third preset quantities can be set according to actual data retrieval needs, and are not specifically limited here.

[0114] Specifically, the diversity score can be determined by the following formula (4): (4) in, Indicates diversity score, Indicates reference primitive, Indicates reference primitive Overall score Represents the resulting primitive set, Indicates reference primitive With the result set of primitives The maximum vector similarity among the reference primitives in the image. This indicates configuration parameters.

[0115] The configuration parameter is a preset balance factor with a value range of 0 to 1, used to control the weight distribution between relevance score and diversity.

[0116] In this way, by selecting a second preset number of reference primitives with the highest comprehensive scores from multiple reference primitives to form a candidate primitive set, and iteratively performing operations based on the comprehensive scores of each candidate primitive and its maximum vector similarity with each primitive in the result primitive set to determine the diversity score, and moving the primitive with the highest diversity score from the candidate primitive set to the result primitive set, until the number of primitives in the result primitive set reaches a third preset number, the final selected multiple reference primitives take into account the degree of content difference between each other on the basis of the relevance of the comprehensive scores. This avoids the reference primitives with high comprehensive scores being concentrated in the same tag path or highly similar content features and thus improving the coverage and diversity of the display results in the tag dimension and content feature dimension, providing more comprehensive reference for annotators.

[0117] In some possible implementations, for each reference graphic element, a preference score for the reference graphic element can be determined based on the user's historical annotation behavior and historical graphic element selection results; a target score for the reference graphic element can be determined based on the comprehensive score and preference score; and the multiple reference graphic elements can be displayed in the annotation interface in descending order of the target scores. Optionally, multiple filtered reference graphic elements can be displayed in the annotation interface in descending order of the target scores.

[0118] In this way, based on the user's historical annotation behavior and historical element selection results, the user's annotation preferences can be determined, such as which tag types they prefer to use and which source items they prefer to use. By introducing preference scoring, elements that match the user's preferences can be ranked higher in the display, improving the user's annotation experience.

[0119] It is understood that for new annotators, there are no associated historical annotation behaviors and historical primitive selection results, so the corresponding preference score is zero. The multiple reference primitives can be displayed in the annotation interface in descending order of the comprehensive score.

[0120] In this embodiment of the disclosure, annotation reference information corresponding to the retrieved data can be generated based on the label name, label path, descriptive text, and attribute text of the target graphic element. Specifically, the label name, label path, descriptive text, and attribute text of the target graphic element can be reused to generate annotation reference information corresponding to the retrieved data.

[0121] Here, the label name and label path of the target graphic element can be used as the label name and label path of the retrieved data, thereby achieving label reuse. The attribute field structure and default values ​​in the attribute text of the target graphic element can be copied to the retrieved data. Users can modify the reused attribute text according to the actual situation of the retrieved data, thereby achieving attribute template reuse. The descriptive text of the target graphic element can be used as a reference template for the description of the retrieved data, thereby achieving descriptive text reuse.

[0122] Optionally, if the target graphic element has a relationship with other graphic elements, such as belonging to the same parent target, the annotation reference information corresponding to the search data can be generated based on the label name, label path, description text, attribute text of the target graphic element and the relationship between the target graphic element and other graphic elements, thereby realizing the reuse of graphic element relationship.

[0123] In some possible implementations, converting the first annotation control structure corresponding to the target primitive into a second annotation control structure that matches the retrieved data includes: When the target graphic element is an image graphic element, the coordinates indicated by the first annotation control structure corresponding to the target graphic element are converted into coordinates that match the size of the retrieved data. Based on the coordinates that match the size of the retrieved data, a second annotation control structure that matches the retrieved data is generated. When the target graphic element is a text graphic element, the start and end character offsets are determined based on the text matching results between the target graphic element and the search data. Based on the first annotation control structure corresponding to the target graphic element and the start and end character offsets, a second annotation control structure matching the search data is generated. When the target graphic element is an audio or video graphic element, based on the sampling rate, frame rate, and time axis offset, the start and end times indicated by the first annotation control structure corresponding to the target graphic element are converted into start and end times that match the time period corresponding to the search data. Based on the start and end times that match the time period corresponding to the search data, a second annotation control structure that matches the search data is generated. When the target primitive is a point cloud primitive, a second annotation control structure matching the retrieved data is generated based on the first annotation control structure corresponding to the target primitive and the coordinate system calibration parameters.

[0124] In the above steps, when the target primitive is an image primitive, the coordinates indicated by the first annotation control structure corresponding to the target primitive are converted into normalized coordinates relative to the image width and height of the retrieved data. Then, according to the image width and height of the retrieved data, the normalized coordinates relative to the image width and height of the retrieved data are converted into coordinates matching the size of the retrieved data. Optionally, when the image of the retrieved data contains cropping, rotation, or scaling information, coordinate mapping can be performed according to the affine transformation matrix in the file metadata. Based on the coordinates matching the size of the retrieved data, a second annotation control structure matching the retrieved data is generated.

[0125] When the target graphic element is a text graphic element, the start and end character offsets are determined based on the text matching results between the target graphic element and the search data, such as string matching results or semantic matching results. Based on the start and end character offsets, the first annotation control structure corresponding to the target graphic element is adjusted to generate a second annotation control structure that matches the search data.

[0126] When the target primitive is an audio / video primitive (audio primitive or video primitive), the first annotation control structure corresponding to the target primitive is obtained, and the first start time and first end time indicated therein are extracted; the first sampling rate, first frame rate, and first time axis offset corresponding to the target primitive are obtained, as well as the second sampling rate, second frame rate, and second time axis offset corresponding to the search data; based on the ratio of the first sampling rate to the second sampling rate, the first start time and the first end time are mapped from the first sampling rate time axis to the second sampling rate time axis to obtain the start time and end time after sampling rate conversion; based on the ratio of the first frame rate to the second frame rate, the start time and end time after sampling rate conversion are corrected by frame rate compensation to obtain the start time and end time after frame rate compensation; based on the difference between the first time axis offset and the second time axis offset, the start time and end time after frame rate compensation are corrected by time axis offset to obtain the target start time and target end time matching the time period corresponding to the search data; based on the target start time and the target end time, a second annotation control structure matching the search data is generated.

[0127] When the target primitive is a point cloud primitive, the first annotation control structure corresponding to the target primitive is obtained, and the first three-dimensional coordinate set of each annotation point indicated therein in the original point cloud coordinate system is extracted; the first coordinate system calibration parameters corresponding to the target primitive and the second coordinate system calibration parameters corresponding to the retrieved data are obtained, wherein the coordinate system calibration parameters include translation vector, rotation matrix and scaling factor; based on the transformation relationship between the first coordinate system calibration parameters and the second coordinate system calibration parameters, rotation correction, translation correction and scaling correction are applied to each three-dimensional coordinate in the first three-dimensional coordinate set in sequence to obtain the second three-dimensional coordinate set of each annotation point in the target point cloud coordinate system; based on the second three-dimensional coordinate set, a second annotation control structure matching the retrieved data is constructed, and a second annotation control structure matching the retrieved data is generated.

[0128] In this way, by converting the coordinates indicated by the first annotation control structure to coordinates matching the size of the retrieved data for image primitives, determining the start and end character offsets based on the text matching results for text primitives and generating matching control structures accordingly, converting the start and end times of audio and video primitives to start and end times matching the time period of the retrieved data based on the sampling rate, frame rate, and time axis offset, and converting point cloud primitives based on coordinate system calibration parameters, the reference primitive annotation control structures under different types of media data can adaptively adapt to the size, text range, time axis, or spatial coordinate system of the current retrieved data. This reduces the frequency of users manually adjusting control coordinates, character offsets, or time start and end positions, thereby improving the smoothness and adaptation accuracy of the annotation control reuse process.

[0129] In some possible implementations, the reference relationship between the retrieved data and the referenced graphic element can be recorded. The reference record may include the retrieved data ID, the referenced graphic element ID, the version number of the referenced graphic element, the reference time, the referrer, and the reference method. The reference method may be, for example, label reuse, description reuse, attribute reuse, or annotation control structure reuse.

[0130] In this way, by recording the reference relationships between retrieved data and referenced features, each annotation result can be associated with its referenced source features and their version information. When the reference features are updated or abnormal, the affected annotation data can be quickly located based on the reference records, facilitating change dissemination notification or review processing and preventing errors from spreading and amplifying in the reference chain. At the same time, the complete reference relationship record provides a traceable reference link for the annotation results, supporting subsequent quality traceability and audit verification of the annotation results, which helps to improve the consistency, traceability, standardization and reproducibility of organizational-level annotation results.

[0131] The foregoing embodiments describe providing auxiliary references during the process of users annotating data through the annotation interface. In other embodiments, auxiliary references can also be provided during the process of users reviewing the annotation results. The auxiliary references provided during the user review process are similar to those provided during the user's data annotation process through the annotation interface, except for the difference in the retrieval trigger event and the retrieval data. For a detailed description, please refer to the foregoing embodiments, which will not be repeated here. In the scenario where users review the annotation results, the corresponding retrieval trigger event may include the creation of a review question, and the corresponding retrieval data may be the annotation results being reviewed.

[0132] In some possible implementations, the method further includes: If labeled data is detected, the review status of the labeled data is determined, and a preset primitive corresponding to the labeled data is generated. Determine the review pass rate, citation popularity, consistency score, correction feedback score, and time validity score corresponding to the preset graphic elements; The quality of the preset graphic elements is determined based on their review pass rate, citation popularity, consistency score, correction feedback score, and time effectiveness score. If the review status is "passed" and the quality of the graphic element is greater than the graphic element quality threshold, the preset graphic element is added to the graphic element library.

[0133] In this step, the new annotations are automatically added to the database after passing the review, and the quality score of the graphic elements is dynamically updated based on the review results, the number of citations, and feedback from manual corrections.

[0134] In the above steps, upon detecting the generation of labeled data, the review status corresponding to the labeled data is determined, and a preset graphic element corresponding to the labeled data is generated. Here, the process of determining the graphic element quality of the preset graphic element is similar to the process of determining the graphic element quality of the reference graphic element described in the previous embodiments. For a detailed description, please refer to the previous embodiments, which will not be repeated here.

[0135] When the review status is "approved" and the element quality is greater than the element quality threshold, the preset element is added to the element library. This ensures that elements with quality less than or equal to the element quality threshold, as well as elements with abnormal review status (e.g., low-quality elements, rejected elements, frozen elements, deleted elements, etc.), are not added to the library. The element quality threshold can be set according to actual data annotation needs and is not specifically limited here.

[0136] Optionally, if a rejected graphic element is manually corrected and re-approved, and its quality is greater than the graphic element quality threshold, it can be added back to the graphic element library. However, the approval rate factor in its graphic element quality will be recorded as a rejection history.

[0137] Optionally, if the review status is passed and the quality of the graphic element is greater than the graphic element quality threshold, check whether the preset graphic element is duplicated with the preset graphic element already stored in the graphic element library. For example, approximate matching can be performed based on media type, tag, or geographical location. If duplicates are found, the preset graphic element is merged and deduplicated when it is added to the graphic element library.

[0138] In this way, by detecting the review status of the labeled data and generating corresponding preset primitives, the quality of primitives is determined based on multiple dimensions such as review pass rate, citation popularity, consistency score, correction feedback score, and time validity score. When the review status is passed and the primitive quality is greater than the primitive quality threshold, the preset primitives are added to the primitive library. This allows high-quality historical labeling results to be automatically converted into searchable and reusable primitive objects after each labeling task is completed, without the need for additional manual sorting and maintenance, thus achieving automated incremental accumulation of the primitive library. At the same time, the multi-dimensional quality scoring mechanism filters new primitives, ensuring that low-quality primitives are not added to the library, while high-quality primitives can be continuously accumulated. This helps the overall quality of the primitive library to gradually improve as labeling tasks progress, providing a more reliable primitive data foundation for retrieval and reuse in subsequent labeling processes.

[0139] In some possible implementations, when a referenced graphic element undergoes minor changes, major changes, or deletion, the retrieval data referencing that graphic element is generated with a status indicating that the reference source has been updated, needs to be re-examined, or is an isolated reference, and the processing is written to the reference audit log.

[0140] Specifically, a primitive reference dependency graph can be constructed. Nodes in this graph represent referenced primitives and their referenced retrieval data, while edges represent reference relationships. Edge attributes include the referenced primitive ID, referenced version number, reference method, reference time, and the person performing the reference. When a referenced primitive undergoes minor changes such as coordinate adjustments or attribute additions, the retrieval data referencing that primitive can be marked as "Reference Source Updated" and a one-click synchronization entry can be provided. When a referenced primitive undergoes significant changes such as label changes, review rejections, or quality scores falling below a threshold, the retrieval data referencing that primitive can be marked as "Requires Re-review." When a referenced primitive is deleted, the retrieval data referencing that primitive can be marked as "Orphaned Reference." Here, changes in the status of primitives and retrieval data are written to the reference audit log, allowing older versions of primitives to still be traced and viewed according to reference records.

[0141] In this way, by constructing a primitive reference dependency graph and recording the referenced primitive and the retrieval data referencing it with nodes, and recording the reference relationship and its attributes with edges, when the referenced primitive is changed or deleted, the retrieval data referencing it is marked as "reference source updated," "requires re-examination," or "isolated reference" according to the change type. The processing is written to the reference audit log, so that the status change of the referenced primitive can be perceived by its associated annotation data. This avoids the continuous use of invalid or corrected annotation information in historical annotation results due to changes in the reference source, while retaining the ability to trace and view the old version of primitives and their reference records, thereby improving the maintainability and audit reproducibility of annotation results in the knowledge evolution process.

[0142] In some possible implementations, the quality change trend of each preset element in the element library can be detected in real time. If the quality score of any preset element continuously decreases within a preset time window and the decrease exceeds a preset threshold, the preset element is removed from the element library, and a review notification is sent to the element library administrator. Optionally, an element reference consistency report can also be generated periodically. This report displays nodes with inconsistent states in the reference chain for auditing by the element library administrator.

[0143] For a clearer illustration of the data annotation process, see [link to documentation]. Figure 3 This is a schematic diagram illustrating a data annotation process provided in an embodiment of this disclosure. Figure 3As shown, a primitive library can be constructed; in response to the detection of a retrieval trigger event, a mixed retrieval of preset primitives in the primitive library is performed to obtain multiple reference primitives; the multiple reference primitives are displayed in the annotation interface, and when it is detected that the user selects a target primitive from the multiple reference primitives, the target primitive is reused to assist the user in annotation processing, thereby realizing the conversion of historical annotation results into real-time reference reuse. Specific steps are described in the foregoing embodiments and will not be repeated here.

[0144] The data annotation method provided in this disclosure pre-converts annotated data into preset primitives carrying structural and content indexes. During the user's data annotation process, when a retrieval trigger event is detected, a mixed retrieval can be performed based on the retrieval data and query information, according to the structural and content indexes. The retrieved reference primitives are displayed, and the target primitives selected by the user are reused to generate annotation reference information and annotation control structures corresponding to the retrieval data. This reduces the frequency of users manually adjusting the annotation control structure, assists users in annotation processing, and effectively improves annotation efficiency. Furthermore, by displaying reference primitives, it helps different annotators to have a consistent understanding and execution of the same tagging system, reduces annotation deviations caused by subjective judgment, ensures the consistency of organizational-level tagging, and improves annotation quality.

[0145] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0146] Based on the same inventive concept, this disclosure also provides a data annotation device corresponding to the data annotation method. Since the principle of the data annotation device in this disclosure is similar to that of the data annotation method described above in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0147] Please see Figure 4 and Figure 5 , Figure 4 This is one of the schematic diagrams of a data annotation device provided in an embodiment of this disclosure. Figure 5 This is a second schematic diagram of a data annotation device provided in an embodiment of this disclosure. For example... Figure 4 As shown in the illustration, the data annotation apparatus 400 provided in this embodiment includes: The retrieval triggering module 401 is used to determine the retrieval data corresponding to the retrieval triggering event and the query information corresponding to the retrieval data in response to the detection of a retrieval triggering event during the process of a user annotating data through the annotation interface. The primitive library acquisition module 402 is used to acquire a primitive library; the primitive library stores multiple preset primitives, which are generated based on labeled data. Each preset primitive is associated with a corresponding structure index and content index. The structure index includes organization number, permission scope, tag code, and tag path. The content index includes descriptive text, attribute text, tag name, media type, geographic location, and content vector. The candidate element retrieval module 403 is used to retrieve multiple candidate elements from the element library based on the query information and the structural index of each preset element. The reference element retrieval module 404 is used to retrieve multiple candidate elements based on the retrieval data, the query information, and the content index of each candidate element, to obtain multiple reference elements. The primitive display module 405 is used to display the plurality of reference primitives in the annotation interface, and when it detects that the user selects a target primitive from the plurality of reference primitives, it generates annotation reference information corresponding to the search data based on the label name, label path, description text and attribute text of the target primitive, and converts the first annotation control structure corresponding to the target primitive into a second annotation control structure that matches the search data; the annotation reference information and the second annotation control structure are used to assist the user in annotating the search data.

[0148] In one optional implementation, the reference primitive retrieval module 404 is specifically used for: Based on the query information and the content index of each candidate graphic element, a search is conducted among multiple candidate graphic elements according to the description text, the attribute text, the tag name, and the media type to obtain multiple first candidate graphic elements; Based on the query information and the content index of each candidate graphic element, a search is performed among multiple candidate graphic elements according to the geographical location to obtain multiple second candidate graphic elements; The retrieved data is content encoded to generate a query vector. Based on the query vector and the content index of each candidate graphic element, a search is performed among multiple candidate graphic elements according to the content vector to obtain multiple third candidate graphic elements. Based on multiple first candidate graphic elements, multiple second candidate graphic elements, and multiple third candidate graphic elements, multiple reference graphic elements are determined.

[0149] In one optional implementation, the geographic location includes latitude and longitude coordinates, address description text, and geocoding; When the reference element retrieval module 404 is used to retrieve multiple candidate elements based on the query information and the content index of each candidate element, according to the geographical location, to obtain multiple second candidate elements, it is specifically used for: Based on the query information and the content index of each candidate graphic element, the positions of multiple candidate graphic elements are compared according to the latitude and longitude coordinates to obtain multiple candidate graphic elements with matching positions. Based on the query information and the content index of each candidate graphic element, a semantic comparison is performed on multiple candidate graphic elements that match the location according to the address description text to obtain multiple candidate graphic elements that match semantically. Based on the query information and the content index of each candidate graphic element, semantic comparison is performed on multiple candidate graphic elements that are semantically matched according to the geocoding to obtain multiple candidate graphic elements that are hierarchically matched. Based on multiple candidate primitives in hierarchical matching, multiple second alternative primitives are determined.

[0150] In one optional implementation, when the graphic element display module 405 is used to display the plurality of reference graphic elements in the annotation interface, it is specifically used for: For each reference primitive, determine the tag matching degree, text similarity, primitive quality, time decay factor, citation popularity, spatial distance weight, and vector similarity corresponding to the reference primitive; The comprehensive score of the reference graphic element is determined based on the tag matching degree, text similarity, graphic element quality, time decay factor, citation popularity, spatial distance weight, and vector similarity corresponding to the reference graphic element. The reference elements are displayed in the annotation interface in descending order of the comprehensive score.

[0151] In an optional implementation, when the primitive display module 405 converts the first annotation control structure corresponding to the target primitive into a second annotation control structure that matches the retrieved data, it is specifically used for: When the target graphic element is an image graphic element, the coordinates indicated by the first annotation control structure corresponding to the target graphic element are converted into coordinates that match the size of the retrieved data. Based on the coordinates that match the size of the retrieved data, a second annotation control structure that matches the retrieved data is generated. When the target graphic element is a text graphic element, the start and end character offsets are determined based on the text matching results between the target graphic element and the search data. Based on the first annotation control structure corresponding to the target graphic element and the start and end character offsets, a second annotation control structure matching the search data is generated. When the target graphic element is an audio or video graphic element, based on the sampling rate, frame rate, and time axis offset, the start and end times indicated by the first annotation control structure corresponding to the target graphic element are converted into start and end times that match the time period corresponding to the search data. Based on the start and end times that match the time period corresponding to the search data, a second annotation control structure that matches the search data is generated. When the target primitive is a point cloud primitive, a second annotation control structure matching the retrieved data is generated based on the first annotation control structure corresponding to the target primitive and the coordinate system calibration parameters.

[0152] In one alternative implementation, such as Figure 5 As shown, the data annotation device 400 further includes a content vector generation module 406, which is used to generate the content vector through the following steps: Based on the media type of the annotation data corresponding to the preset graphic element, feature extraction is performed on the annotation data corresponding to the preset graphic element to obtain a modal feature vector; Based on the label codes and label paths corresponding to the preset graphic elements, a label path vector is generated; The content vector is generated based on the modal feature vector and the tag path vector.

[0153] In one alternative implementation, such as Figure 5 As shown, the data annotation device 400 further includes a primitive addition module 407, which is used for: If labeled data is detected, the review status of the labeled data is determined, and a preset primitive corresponding to the labeled data is generated. Determine the review pass rate, citation popularity, consistency score, correction feedback score, and time validity score corresponding to the preset graphic elements; The quality of the preset graphic elements is determined based on their review pass rate, citation popularity, consistency score, correction feedback score, and time effectiveness score. If the review status is "passed" and the quality of the graphic element is greater than the graphic element quality threshold, the preset graphic element is added to the graphic element library.

[0154] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0155] The data annotation apparatus provided in this disclosure pre-converts annotated data into preset primitives carrying structural and content indexes. During the user's data annotation process, when a retrieval trigger event is detected, a mixed retrieval can be performed based on the retrieval data and query information, according to the structural and content indexes. The retrieved reference primitives are displayed, and the reference primitives are reused according to the target primitives selected by the user to generate annotation reference information and annotation control structures corresponding to the retrieval data. This reduces the frequency of manual adjustment of annotation control structures by the user, assists the user's annotation process, and effectively improves annotation efficiency. Furthermore, by displaying reference primitives, it helps different annotators to have a consistent understanding and execution of the same tagging system, reduces annotation deviations caused by subjective judgment, ensures the consistency of organizational-level tagging, and improves annotation quality.

[0156] Corresponding to Figure 1 In addition to the data annotation method in this disclosure, this embodiment also provides a computer device 600, such as... Figure 6 The diagram shown is a structural schematic of a computer device 600 provided in an embodiment of this disclosure, comprising: The processor 610, memory 620, and bus 630 are included. The memory 620 is used to store execution instructions and includes main memory 621 and external memory 622. The main memory 621, also known as internal memory, is used to temporarily store the operation data in the processor 610 and the data exchanged with external memory 622 such as hard disk. The processor 610 exchanges data with external memory 622 through main memory 621.

[0157] In this embodiment, the memory 620 is specifically used to store application code that executes the solution of this application, and its execution is controlled by the processor 610. That is, when the computer device 600 is running, the processor 610 communicates with the memory 620 through the bus 630, or the processor 610 communicates with the memory 620 through other means, so that the processor 610 executes the application code stored in the memory 620, and then executes the steps of the data annotation method described in any of the foregoing embodiments.

[0158] The memory 620 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0159] Processor 610 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0160] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the computer device 600. In other embodiments of this application, the computer device 600 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0161] Furthermore, this disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the data annotation method described in the above-described method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0162] This disclosure also provides a computer program product, which stores a computer program. When the computer program is run by a processor, it executes the steps of the data annotation method provided in any of the above embodiments of this disclosure. For details, please refer to the above method embodiments, which will not be repeated here.

[0163] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium, which can be a volatile or non-volatile computer-readable storage medium. In another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0164] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices and apparatuses described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0165] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0167] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A data annotation method, characterized in that, The method includes: During the process of a user annotating data through the annotation interface, in response to the detection of a retrieval trigger event, the retrieval data corresponding to the retrieval trigger event and the query information corresponding to the retrieval data are determined. Obtain a primitive library; the primitive library stores multiple preset primitives, which are generated based on labeled data. Each preset primitive is associated with a corresponding structure index and content index. The structure index includes organization number, permission scope, tag code, and tag path. The content index includes descriptive text, attribute text, tag name, media type, geographic location, and content vector. Based on the query information and the structural index of each preset graphic element, a search is performed in the graphic element library to obtain multiple candidate graphic elements; Based on the retrieval data, the query information, and the content index of each candidate graphic element, a retrieval is performed among multiple candidate graphic elements to obtain multiple reference graphic elements; The annotation interface displays the multiple reference graphic elements. When the user selects a target graphic element from the multiple reference graphic elements, the annotation reference information corresponding to the search data is generated based on the label name, label path, description text, and attribute text of the target graphic element. The first annotation control structure corresponding to the target graphic element is converted into a second annotation control structure that matches the search data. The annotation reference information and the second annotation control structure are used to assist the user in annotating the search data.

2. The method according to claim 1, characterized in that, The process involves searching among multiple candidate graphemes based on the retrieval data, the query information, and the content indexes of each candidate grapheme to obtain multiple reference graphemes, including: Based on the query information and the content index of each candidate graphic element, a search is conducted among multiple candidate graphic elements according to the description text, the attribute text, the tag name, and the media type to obtain multiple first candidate graphic elements; Based on the query information and the content index of each candidate graphic element, a search is performed among multiple candidate graphic elements according to the geographical location to obtain multiple second candidate graphic elements; The retrieved data is content encoded to generate a query vector. Based on the query vector and the content index of each candidate graphic element, a search is performed among multiple candidate graphic elements according to the content vector to obtain multiple third candidate graphic elements. Based on multiple first candidate graphic elements, multiple second candidate graphic elements, and multiple third candidate graphic elements, multiple reference graphic elements are determined.

3. The method according to claim 2, characterized in that, The geographic location includes latitude and longitude coordinates, address description text, and geocoding; Based on the query information and the content index of each candidate graphic element, a search is performed among multiple candidate graphic elements according to the geographical location to obtain multiple second candidate graphic elements, including: Based on the query information and the content index of each candidate graphic element, the positions of multiple candidate graphic elements are compared according to the latitude and longitude coordinates to obtain multiple candidate graphic elements with matching positions. Based on the query information and the content index of each candidate graphic element, a semantic comparison is performed on multiple candidate graphic elements that match the location according to the address description text to obtain multiple candidate graphic elements that match semantically. Based on the query information and the content index of each candidate graphic element, semantic comparison is performed on multiple candidate graphic elements that are semantically matched according to the geocoding to obtain multiple candidate graphic elements that are hierarchically matched. Based on multiple candidate primitives in hierarchical matching, multiple second alternative primitives are determined.

4. The method according to claim 1, characterized in that, The display of the plurality of reference elements in the annotation interface includes: For each reference primitive, determine the tag matching degree, text similarity, primitive quality, time decay factor, citation popularity, spatial distance weight, and vector similarity corresponding to the reference primitive; The comprehensive score of the reference graphic element is determined based on the tag matching degree, text similarity, graphic element quality, time decay factor, citation popularity, spatial distance weight, and vector similarity corresponding to the reference graphic element. The reference elements are displayed in the annotation interface in descending order of the comprehensive score.

5. The method according to claim 1, characterized in that, The step of converting the first annotation control structure corresponding to the target graphic element into a second annotation control structure that matches the retrieved data includes: When the target graphic element is an image graphic element, the coordinates indicated by the first annotation control structure corresponding to the target graphic element are converted into coordinates that match the size of the retrieved data. Based on the coordinates that match the size of the retrieved data, a second annotation control structure that matches the retrieved data is generated. When the target graphic element is a text graphic element, the start and end character offsets are determined based on the text matching results between the target graphic element and the search data. Based on the first annotation control structure corresponding to the target graphic element and the start and end character offsets, a second annotation control structure matching the search data is generated. When the target graphic element is an audio or video graphic element, based on the sampling rate, frame rate, and time axis offset, the start and end times indicated by the first annotation control structure corresponding to the target graphic element are converted into start and end times that match the time period corresponding to the search data. Based on the start and end times that match the time period corresponding to the search data, a second annotation control structure that matches the search data is generated. When the target primitive is a point cloud primitive, a second annotation control structure matching the retrieved data is generated based on the first annotation control structure corresponding to the target primitive and the coordinate system calibration parameters.

6. The method according to claim 1, characterized in that, The content vector is generated through the following steps: Based on the media type of the annotation data corresponding to the preset graphic element, feature extraction is performed on the annotation data corresponding to the preset graphic element to obtain a modal feature vector; Based on the label codes and label paths corresponding to the preset graphic elements, a label path vector is generated; The content vector is generated based on the modal feature vector and the tag path vector.

7. The method according to claim 1, characterized in that, The method further includes: If labeled data is detected, the review status of the labeled data is determined, and a preset primitive corresponding to the labeled data is generated. Determine the review pass rate, citation popularity, consistency score, correction feedback score, and time validity score corresponding to the preset graphic elements; The quality of the preset graphic elements is determined based on their review pass rate, citation popularity, consistency score, correction feedback score, and time validity score. If the review status is "passed" and the quality of the graphic element is greater than the graphic element quality threshold, the preset graphic element is added to the graphic element library.

8. A data annotation device, characterized in that, The device includes: The retrieval triggering module is used to determine the retrieval data corresponding to the retrieval triggering event and the query information corresponding to the retrieval data in response to the detection of a retrieval triggering event during the process of a user annotating data through the annotation interface. The primitive library acquisition module is used to acquire a primitive library; the primitive library stores multiple preset primitives, which are generated based on labeled data. Each preset primitive is associated with a corresponding structure index and content index. The structure index includes organization number, permission scope, tag code, and tag path. The content index includes descriptive text, attribute text, tag name, media type, geographic location, and content vector. The candidate element retrieval module is used to search the element library based on the query information and the structural index of each preset element to obtain multiple candidate elements. The reference element retrieval module is used to retrieve multiple candidate elements based on the retrieval data, the query information, and the content index of each candidate element, to obtain multiple reference elements. The primitive display module is used to display the multiple reference primitives in the annotation interface, and when it detects that the user selects a target primitive from the multiple reference primitives, it generates annotation reference information corresponding to the search data based on the label name, label path, description text and attribute text of the target primitive, and converts the first annotation control structure corresponding to the target primitive into a second annotation control structure that matches the search data; the annotation reference information and the second annotation control structure are used to assist the user in annotating the search data.

9. A computer device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the data annotation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the data annotation method as described in any one of claims 1 to 7.

Images