Method, device, storage medium and electronic device for automatic labeling
By automatically generating annotation prompts from large models, the problem of low efficiency and high cost of traditional manual annotation is solved. It enables rapid production of intelligent capabilities and personalized task adaptation, thereby improving annotation efficiency and quality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
- Filing Date
- 2026-04-08
- Publication Date
- 2026-07-10
AI Technical Summary
Traditional manual annotation methods are inefficient and costly, making it difficult to achieve a transformation to human-machine collaborative work. Customized task development is costly and time-consuming, and the reusability of repeated capability development is weak. The links are weakly coupled and the links are independent.
Based on the understanding and code generation capabilities of large models, intelligent annotation capabilities are automatically generated. Annotation prompts are automatically generated through annotation paradigms, enabling large-scale capability production and personalized task adaptation. This includes inputting annotation paradigms into the model to generate prompts, responding to user requests to output annotation services, and adjusting prompts to adapt to different task requirements.
It has improved the production speed of intelligent capabilities, solved the problems of high cost and long cycle of customized task development, realized large-scale capability production and personalized task adaptation, and improved annotation efficiency and quality.
Smart Images
Figure CN122366352A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to computer technology, and more particularly to a method, apparatus, storage medium, and electronic device for automatic annotation. Background Technology
[0002] The data annotation industry is undergoing a revolutionary transformation from labor-intensive to technology-intensive. Traditional manual annotation methods face efficiency bottlenecks and cost pressures, while modern data annotation has evolved into intelligent systems with deep algorithmic involvement and human-machine collaboration. Previously, annotation tools were developed entirely by algorithms, requiring customized development for each task, resulting in high startup costs and long cycles, making it difficult to achieve a comprehensive human-machine collaborative workflow. This solution, based on an annotation paradigm, leverages the understanding of large models and code generation capabilities to automatically generate intelligent annotation capabilities, thereby improving the speed of intelligent capability production. Summary of the Invention
[0003] The purpose of the embodiments in this specification is to provide a method, apparatus, storage medium, and electronic device for automatic annotation.
[0004] This specification provides a method for automatic annotation. Based on an annotation paradigm, it leverages the understanding of large models and code generation capabilities to automatically generate intelligent annotation capabilities, thereby improving the speed of intelligent capability production. This method can address issues such as high startup costs and long cycles, repetitive capability development, weak reusability, weak link coupling, and independent stages in customized task development. By automatically generating annotation prompts based on the annotation paradigm, the delivery side can autonomously generate debugging annotation prompts as needed and select suitable large models, enabling large-scale capability production and personalized task adaptation. The method includes: Obtain the annotation paradigm of the target annotation task input by the user, input the annotation paradigm into the first model, and obtain the annotation prompt information output by the first model. The annotation paradigm includes the case data to be annotated and the annotation task description information. In response to the user's intelligent capability generation request for the annotation prompt information, the first model outputs the annotation service corresponding to the annotation prompt information. The annotation service is used to automatically annotate the data to be annotated. The annotation service includes the annotation prompt information and code information for providing automatic annotation capabilities for the target annotation task based on the annotation prompt information.
[0005] Furthermore, the annotation paradigm also includes annotation task rule information, which includes at least one of annotation background, annotation steps, and annotation requirements.
[0006] Furthermore, the annotation paradigm also includes annotation titles for the data to be annotated.
[0007] Furthermore, the annotation paradigm also includes annotation template information about the data to be annotated and its corresponding annotations.
[0008] Furthermore, the method also includes: Obtain the modification instruction information input by the user in response to the annotation prompt information, and adjust the annotation prompt information according to the modification instruction information.
[0009] Furthermore, the method also includes: Input the annotation prompt information and the test data to be annotated into the second model to obtain the predicted annotation information corresponding to the test data output by the second model; Based on the predicted annotation information, obtain annotation evaluation information about the annotation prompt information.
[0010] Furthermore, the method also includes: The annotation prompt information is adjusted based on the annotation evaluation information.
[0011] Furthermore, adjusting the annotation prompt information based on the annotation evaluation information includes: The annotation evaluation information is input into the first model to obtain the adjusted annotation prompt information output by the first model based on the adjusted annotation paradigm.
[0012] Furthermore, adjusting the annotation prompt information based on the annotation evaluation information includes: In response to the user adjusting the annotation paradigm based on the annotation evaluation information, the adjusted annotation paradigm is re-input into the first model to obtain the adjusted annotation prompt information output by the first model based on the adjusted annotation paradigm.
[0013] Furthermore, the code information includes first code information for inputting the annotation prompt information and the data to be annotated into the third model and obtaining the annotation information output by the third model.
[0014] Furthermore, the code information also includes second code information for performing target processing on the data to be labeled and / or the labeling information, wherein the target processing includes at least one of data format cleaning, data table processing, and anomaly detection processing.
[0015] Furthermore, the code information also includes third code information used to retrieve the annotation information and determine the authenticity of the annotation information based on the retrieval results.
[0016] Furthermore, the method also includes: Obtain multiple annotation information that meet the preset conditions for authenticity, and adjust the annotation prompt information according to the multiple annotation information and their corresponding data to be annotated; This enables the first model to output the latest annotation service corresponding to the adjusted annotation prompt information.
[0017] This specification also provides an apparatus for automatic labeling, comprising: The annotation prompt information module is used to obtain the annotation paradigm of the target annotation task input by the user, input the annotation paradigm into the first model, and obtain the annotation prompt information output by the first model. The annotation paradigm includes the case data to be annotated and the annotation task description information. The annotation service module is used to respond to the intelligent capability generation request initiated by the user in response to the annotation prompt information, so that the first model outputs the annotation service corresponding to the annotation prompt information. The annotation service is used to automatically annotate the data to be annotated. The annotation service includes the annotation prompt information and code information for providing automatic annotation capabilities for the target annotation task based on the annotation prompt information.
[0018] This specification also provides a storage medium storing a computer program adapted to be loaded by a processor and to execute the steps of the method described above.
[0019] This specification also provides an electronic device, including a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method described above.
[0020] This specification also provides a computer program product that stores at least one instruction, characterized in that the at least one instruction, when executed by a processor, implements the steps of the above-described method.
[0021] According to the embodiments of this specification, based on the annotation paradigm, the intelligent annotation capability is automatically generated through the understanding of large models and code generation capabilities, thereby improving the production speed of intelligent capabilities. It can be used to solve the problems of customized task development, such as high start-up costs and long cycles, repetitive capability development, weak reusability, weak link coupling, and independent links. By automatically generating annotation prompt information based on the annotation paradigm, the delivery side can independently generate debugging annotation prompt information as needed and select the appropriate large model, which can realize large-scale capability production and personalized task adaptation. Attached Figure Description
[0022] Figure 1 A flowchart illustrating a method for automatic annotation provided in an embodiment of this specification; Figure 2 A flowchart illustrating a method for automatic annotation provided in an embodiment of this specification; Figure 3 A schematic diagram of a device for automatic labeling provided in an embodiment of this specification; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.
[0024] Please see Figure 1 This is a flowchart illustrating a method for automatic annotation provided in an embodiment of this specification. In this embodiment, the method for automatic annotation is applied to the automatic annotation apparatus (hereinafter referred to as the "automatic annotation apparatus") or an electronic device equipped with an automatic annotation apparatus as described in this embodiment. The following will focus on... Figure 1 The process shown will be described in detail. The method for automatic annotation may specifically include the following steps: S102, obtain the annotation paradigm of the target annotation task input by the user, input the annotation paradigm into the first model, and obtain the annotation prompt information output by the first model, wherein the annotation paradigm includes the case data to be annotated and the annotation task description information.
[0025] In some embodiments, the user can be the project manager for the annotation task, responsible for managing project progress and delivery quality. In some embodiments, an annotation task refers to the task of annotating data to be annotated to form high-quality annotated data that can be used for model training / fine-tuning / alignment. In some embodiments, an annotation paradigm refers to a unified and standardized annotation method, format, rules, and process system in data annotation tasks, used to ensure that the annotation results produced by different annotators are consistent, standardized, and can be directly used by the model.
[0026] In some embodiments, the annotation paradigm includes case data to be annotated and annotation task description information. The case data to be annotated can be a newly created dataset by the user, or it can be selected by the user from an existing dataset (e.g., by searching for the corresponding dataset ID). In some embodiments, the annotation task description information is used to describe the target annotation task. It is a text that provides a complete and clear explanation of the content of the annotation task, allowing the annotator to accurately understand what to do, how to do it, and to what extent. For example, the annotation task description information is "This is an image classification task used to identify products in images." In some embodiments, the annotation paradigm is input into a first model. The first model includes, but is not limited to, any large language model with any model structure and any model parameters. This example embodiment does not impose any special limitations on this. In some embodiments, the first model understands and reasones about the annotation paradigm, and then generates and outputs corresponding annotation prompt information, changing from the original sequential development to parallel development, which greatly improves the development speed. Here, the annotation prompt information refers to a piece of instructional text directly sent to the model during the data annotation process to clearly tell the model how to annotate, with the purpose of letting the model know how to annotate and what the annotation should look like.
[0027] S104, in response to the intelligent capability generation request initiated by the user for the annotation prompt information, the first model outputs the annotation service corresponding to the annotation prompt information, wherein the annotation service is used to automatically annotate the data to be annotated, and the annotation service includes the annotation prompt information and code information for providing automatic annotation capabilities for the target annotation task based on the annotation prompt information.
[0028] In some embodiments, in response to a user's intelligent capability generation request for annotation prompts, the first model outputs an annotation service for automatically annotating the data to be annotated. The data to be annotated includes, but is not limited to, any data that needs to be annotated; this example embodiment does not specifically limit this, meaning the data to be annotated is distinct from the case data to be annotated, and there is no correlation between the two. In some embodiments, the annotation service includes annotation prompts and code information for providing automatic annotation capabilities for the target annotation task based on the annotation prompts. That is, the code information allows the data to be annotated and the annotation prompts to be input into the corresponding model and obtain the annotation information output by the model, thereby achieving automatic annotation of the data to be annotated. The model includes, but is not limited to, any large language model with any model structure and any model parameters. The model can be the first model itself, or it can be any other large language model besides the first model used to perform automatic annotation. In some embodiments, the code information in the annotation service is hidden from the user; by running (or compiling and running) this code information, the user can be provided with automatic annotation capabilities for the target annotation task. In some embodiments, the annotation service output by the first model has intelligent capabilities. Intelligent capabilities refer to the introduction of machine algorithms in the annotation process, thereby improving the annotation quality and efficiency for annotators. In other words, the annotation service is an intelligent tool for automatic annotation.
[0029] According to the embodiments of this specification, based on the annotation paradigm, the intelligent annotation capability is automatically generated through the understanding of large models and code generation capabilities, thereby improving the production speed of intelligent capabilities. It can be used to solve the problems of customized task development, such as high start-up costs and long cycles, repetitive capability development, weak reusability, weak link coupling, and independent links. By automatically generating annotation prompt information based on the annotation paradigm, the delivery side can independently generate debugging annotation prompt information as needed and select the appropriate large model, which can realize large-scale capability production and personalized task adaptation.
[0030] In some embodiments, the annotation paradigm further includes annotation task rule information, which includes at least one of annotation background, annotation steps, and annotation requirements. In some embodiments, annotation task rule information refers to annotation instructions for annotators, who perform annotations according to these instructions. During the data annotation process, the annotation task rule information is used to provide unified, clear, and executable regulations for at least one of the annotation background, annotation steps, and annotation requirements, ensuring that different annotators produce consistent, accurate, and high-quality annotated data. The annotation rules explain why this annotation is being performed, the source and purpose of the data to be annotated, which stage of the large model the annotation results will serve (training, fine-tuning, evaluation, alignment, security audit, etc.), and what business problem this annotation aims to solve. The annotation steps provide a procedural and sequential description of the annotation operation, clarifying what annotators should do and in what order from opening the task to submitting the results; it is a directly executable operational process. The annotation requirements are unified constraints on the quality, specifications, boundaries, and prohibitions of the annotation results, clarifying what is qualified, what is unqualified, what cannot be annotated, and what must be annotated; it serves as the basis for judging whether the annotation meets the standards.
[0031] In some embodiments, the annotation paradigm further includes annotation titles for the data to be annotated. In some embodiments, an annotation title refers to a specific question explicitly posed to the annotator in the annotation task, requiring judgment / filling / selection of the data to be annotated. It is the smallest task unit that the annotator directly operates on and answers. For example, if the target annotation task is to annotate the product category options for e-commerce, the corresponding annotation title could be "Product Category".
[0032] In some embodiments, the annotation paradigm further includes annotation template information about the data to be annotated and its corresponding annotations. In some embodiments, the annotation template information is a pre-designed structured framework and form in the indicator annotation task to standardize and unify annotation behavior. It contains all interactive elements associated with the data to be annotated and its corresponding annotation information in this annotation task. It is the operation interface and filling template directly used by the annotators when actually performing annotation, and is used to standardize the annotation task and ensure that the annotation process is efficient and the results are consistent.
[0033] In some embodiments, the method further includes: obtaining modification instruction information input by the user for the annotation prompt information, and adjusting the annotation prompt information according to the modification instruction information. In some embodiments, if the user is not satisfied with the annotation prompt information output by the first model, they can input modification instruction information for the annotation prompt information. The modification instruction information is used to indicate how to modify the annotation prompt information, and then the annotation prompt information is adjusted according to the modification instruction information. For example, by performing semantic recognition on the modification instruction information, the original annotation prompt information is adjusted based on the semantic recognition result to obtain the adjusted annotation prompt information. Alternatively, the modification instruction information is input into the first model, so that the first model adjusts based on the original annotation prompt information and outputs the adjusted annotation prompt information. That is, when the user inputs the modification instruction information, the first model automatically adjusts based on the original annotation prompt information, eliminating the need for repeated development.
[0034] In some embodiments, the method further includes: inputting the annotation prompt information and the test data to be annotated into a second model to obtain predicted annotation information corresponding to the test data output by the second model; and obtaining annotation evaluation information about the annotation prompt information based on the predicted annotation information. In some embodiments, the annotation prompt information and the test data to be annotated are input into a second model, wherein the second model includes, but is not limited to, a large language model with any model structure and any model parameters. The model can be the first model itself, or it can be any other large language model used to test automatic annotation, except for the first model. The test data to be annotated and the case data to be annotated can be the same data, or they can be different data. This example embodiment does not impose any special limitations on this. In some embodiments, the second model outputs predicted annotation information corresponding to the test data. Based on the predicted annotation information, annotation evaluation information about the annotation prompts can be obtained. The annotation evaluation information is used to evaluate the accuracy and coverage of the annotation capability of the annotation prompts. This evaluation can be performed manually by the user based on the predicted annotation information, and the user-inputted annotation evaluation information about the annotation prompts can be obtained. Alternatively, the test data and the corresponding predicted annotation information can be input into a preset evaluation model, which evaluates the annotation capability of the annotation prompts based on the predicted annotation information and outputs the annotation evaluation information about the annotation prompts. The evaluation model is a different model from the second model. The evaluation model includes, but is not limited to, large language models with any model structure and any model parameters. For example, the evaluation model will retrieve the test data during the evaluation process, match the retrieval results with the predicted annotation information, and evaluate the annotation capability of the annotation prompts based on the matching results. In some embodiments, it can be determined whether the previous annotation prompts need to be adjusted based on the annotation evaluation information. For example, the user can determine whether the previous annotation prompts need to be adjusted based on the annotation evaluation information. Alternatively, it can be determined automatically based on the recognition or comparison results by performing semantic recognition, keyword recognition, or numerical threshold comparison on the annotation evaluation information.
[0035] In some embodiments, the method further includes: adjusting the annotation prompt information based on the annotation evaluation information. In some embodiments, a user can manually adjust the annotation prompt information based on the annotation evaluation information. For example, a user can input modification instructions for the original annotation prompt information based on the annotation evaluation information, and adjust the original annotation prompt information according to the modification instructions to obtain the adjusted annotation prompt information. In some embodiments, the annotation prompt information can also be automatically adjusted based on the annotation evaluation information to obtain the adjusted annotation prompt information. For example, by performing semantic recognition on the annotation evaluation information, determining at least one problematic text information in the original annotation prompt information based on the semantic recognition results, and determining modification instructions for indicating how to modify the at least one text information, and then adjusting the original annotation prompt information according to the modification instructions.
[0036] In some embodiments, adjusting the annotation prompt information based on the annotation evaluation information includes: inputting the annotation evaluation information into the first model to obtain adjusted annotation prompt information output by the first model based on the adjusted annotation paradigm. In some embodiments, the annotation evaluation information can be input into the first model, causing the first model to infer problems existing in the original annotation prompt information based on the annotation evaluation information, determine modification instruction information for indicating how to modify the existing problems based on the inferred problems, then adjust the original annotation prompt information according to the modification instruction information, and output the adjusted annotation prompt information.
[0037] In some embodiments, the method further includes: responding to the user adjusting the annotation paradigm based on the annotation evaluation information, re-inputting the adjusted annotation paradigm into the first model to obtain adjusted annotation prompt information output by the first model based on the adjusted annotation paradigm. In some embodiments, the user can adjust the original annotation paradigm by referring to the annotation evaluation information, and then re-input the adjusted annotation paradigm into the first model. The first model will adjust the original annotation prompt information based on the adjusted annotation paradigm and output the adjusted annotation prompt information.
[0038] In some embodiments, the code information includes first code information for inputting the annotation prompt information and the data to be annotated into a third model and obtaining the annotation information output by the third model. In some embodiments, the first code information can be used to input the annotation prompt information and the data to be annotated into a third model and obtain the annotation information corresponding to the data to be annotated output by the third model. The third model includes, but is not limited to, a large language model with any model structure and any model parameters. The model can be the first model itself, or it can be any other large language model used to perform automatic annotation besides the first model.
[0039] In some embodiments, the code information further includes second code information for performing target processing on the data to be labeled and / or the labeling information, wherein the target processing includes at least one of data format cleaning, data table processing, and anomaly detection processing. In some embodiments, the second code information can be used to perform target processing on the data to be labeled before inputting it into the third model, and then input the processed data to be labeled into the third model. In some embodiments, the second code information can be used to perform target processing on the labeling information after obtaining the labeling information corresponding to the data to be labeled output by the third model. In some embodiments, target processing includes, but is not limited to, at least one of data format cleaning, data table processing, and anomaly detection processing. This example embodiment does not impose any special limitations on this. Data format cleaning refers to structuring, standardizing, and denoising the data to be labeled before inputting it into the third model, and organizing messy, inconsistent, and unusable data into a format that the model can stably recognize and process. Data table processing refers to performing structured parsing, field alignment, row and column regularization, merging, and splitting on the structured table-formatted data to be labeled and / or labeling information to meet the input requirements of the third model or downstream use. Anomaly detection processing refers to automatically identifying data that does not conform to rules, exceeds the range, has logical contradictions, or is obviously erroneous before inputting the data to be labeled into the third model or after obtaining the labeling information output by the third model, and correcting, filtering, marking, or discarding it.
[0040] In some embodiments, the code information further includes third code information for retrieving the annotation information and determining the authenticity of the annotation information based on the retrieval results. In some embodiments, the third code information enables the retrieval (search engine retrieval or knowledge base retrieval) of the annotation information corresponding to the data to be annotated output by the third model after obtaining the annotation information, and the determination of the authenticity of the annotation information based on the retrieval results. Authenticity can be a text, such as "authentic" or "unauthentic," or it can be a specific numerical value representing the degree of authenticity of the annotation information. In some embodiments, the annotation service displays the authenticity of the annotation information to the user after completing automatic annotation, or the annotation service determines whether to display the annotation information to the user based on the authenticity of the annotation information after completing automatic annotation. For example, the annotation information is only displayed to the user if its authenticity is "authentic," or only if the authenticity of the annotation information is greater than or equal to a preset threshold.
[0041] In some embodiments, the method further includes: obtaining multiple annotation information whose authenticity meets preset conditions; adjusting the annotation prompt information according to the multiple annotation information and their corresponding unannotated data; and causing the first model to output the latest annotation service corresponding to the adjusted annotation prompt information. In some embodiments, after the annotation service completes multiple automatic annotations, multiple annotation information whose authenticity meets preset conditions are obtained from the multiple automatic annotations. The preset conditions may be authenticity as "authentic", or authenticity greater than or equal to a preset threshold. This example embodiment does not specifically limit this. In some embodiments, adjusting the annotation prompt information according to the multiple annotation information and their corresponding unannotated data may involve obtaining feature information of the unannotated data, performing semantic recognition on the annotation information, determining several texts in the original annotation prompt information that have authenticity risks and the corresponding modification methods of the several texts based on the semantic recognition results and the feature information of the unannotated data, then adjusting the several texts based on the modification methods to obtain the adjusted annotation prompt information, and then re-inputting the adjusted annotation prompt information into the first model to obtain the latest annotation service output by the first model based on the adjusted annotation prompt information. In some embodiments, the multiple annotation information and their corresponding data to be annotated are directly input into the first model, so that the first model adjusts the previous annotation prompt information and outputs the latest annotation service based on the adjusted annotation prompt information.
[0042] Figure 2 This is a flowchart illustrating a method for automatic annotation provided in an embodiment of this specification.
[0043] like Figure 2As shown, in the input phase, the PA (Project Manager for Labeling Tasks) inputs templates, labeling rules, case data, task descriptions, etc. In the Prompt generation phase, the large model understands and reflects on the input data to generate an automatically labeled Prompt. In the review and modification phase, the PA reviews the Prompt. If there are any problems, the PA can choose to modify the large model or manually modify the Prompt. In the model testing phase, the PA obtains the test accuracy corresponding to the Prompt. If the test accuracy is lower than or equal to 80%, the PA needs to return to the review and modification phase to modify the Prompt. If the test accuracy is higher than 80%, the PA enters the iteration or completion phase. If the PA determines that the Prompt meets the requirements, the PA can click "Complete" to generate the labeling service (i.e., intelligent tool) corresponding to the Prompt.
[0044] Figure 3 This is a schematic diagram of a device for automatic annotation provided in an embodiment of this specification. This device (hereinafter referred to as "automatic annotation device 1") can be implemented as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the automatic annotation device 1 includes an annotation prompt information module 11 and an annotation service module 12.
[0045] The annotation prompt information module 11 is used to obtain the annotation paradigm of the target annotation task input by the user, input the annotation paradigm into the first model, and obtain the annotation prompt information output by the first model. The annotation paradigm includes the case data to be annotated and the annotation task description information. The annotation service module 12 is used to respond to the intelligent capability generation request initiated by the user for the annotation prompt information, so that the first model outputs the annotation service corresponding to the annotation prompt information. The annotation service is used to automatically annotate the data to be annotated. The annotation service includes the annotation prompt information and code information for providing automatic annotation capabilities for the target annotation task based on the annotation prompt information.
[0046] In some embodiments, the annotation paradigm further includes annotation task rule information, which includes at least one of annotation background, annotation steps, and annotation requirements.
[0047] In some embodiments, the annotation paradigm also includes annotation titles for the data to be annotated.
[0048] In some embodiments, the annotation paradigm also includes annotation template information about the data to be annotated and its corresponding annotations.
[0049] In some embodiments, the automatic annotation device 1 is further configured to: obtain modification instruction information input by the user for the annotation prompt information, and adjust the annotation prompt information according to the modification instruction information.
[0050] In some embodiments, the automatic annotation device 1 is further configured to: input the annotation prompt information and the test data to be annotated into a second model to obtain the predicted annotation information corresponding to the test data output by the second model; and obtain annotation evaluation information about the annotation prompt information based on the predicted annotation information.
[0051] In some embodiments, the automatic annotation device 1 is further configured to: adjust the annotation prompt information based on the annotation evaluation information.
[0052] In some embodiments, adjusting the annotation prompt information based on the annotation evaluation information includes: inputting the annotation evaluation information into the first model to obtain the adjusted annotation prompt information output by the first model based on the adjusted annotation paradigm.
[0053] In some embodiments, adjusting the annotation prompt information based on the annotation evaluation information includes: in response to the user adjusting the annotation paradigm based on the annotation evaluation information, re-inputting the adjusted annotation paradigm into the first model to obtain the adjusted annotation prompt information output by the first model based on the adjusted annotation paradigm.
[0054] In some embodiments, the code information includes first code information for inputting the annotation prompt information and the data to be annotated into a third model and obtaining the annotation information output by the third model.
[0055] In some embodiments, the code information further includes second code information for target processing of the data to be labeled and / or the labeling information, wherein the target processing includes at least one of data format cleaning, data table processing, and anomaly detection processing.
[0056] In some embodiments, the code information further includes third code information for retrieving the annotation information and determining the authenticity of the annotation information based on the retrieval results.
[0057] In some embodiments, the automatic annotation device 1 is further configured to: obtain multiple annotation information whose authenticity meets preset conditions, adjust the annotation prompt information according to the multiple annotation information and their corresponding data to be annotated, so that the first model outputs the latest annotation service corresponding to the adjusted annotation prompt information.
[0058] The above-described apparatus embodiments correspond to the aforementioned method embodiments. For detailed descriptions, please refer to the description in the method embodiments section; further details will not be repeated here. The apparatus embodiments are derived from the corresponding method embodiments and have the same technical effects. For detailed descriptions, please refer to the corresponding method embodiments.
[0059] This specification also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in this specification.
[0060] This specification also provides a computer program product that stores at least one instruction, which is loaded by the processor and executes the method described in this specification embodiment.
[0061] This specification also provides an electronic device, including a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and execute the method described in the embodiments of this specification.
[0062] The embodiments in this specification also provide Figure 4 The diagram shows the structure of the electronic device. Figure 4 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the above method.
[0063] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.
[0064] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0065] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0066] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0067] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0068] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0069] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0070] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0071] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.
Claims
1. A method for automatic annotation, comprising: Obtain the annotation paradigm of the target annotation task input by the user, input the annotation paradigm into the first model, and obtain the annotation prompt information output by the first model. The annotation paradigm includes the case data to be annotated and the annotation task description information. In response to the user's intelligent capability generation request for the annotation prompt information, the first model outputs the annotation service corresponding to the annotation prompt information. The annotation service is used to automatically annotate the data to be annotated. The annotation service includes the annotation prompt information and code information for providing automatic annotation capabilities for the target annotation task based on the annotation prompt information.
2. The method according to claim 1, wherein the annotation paradigm further includes annotation task rule information, the annotation task rule information including at least one of annotation background, annotation steps and annotation requirements.
3. The method according to claim 1, wherein the annotation paradigm further includes annotation titles for the data to be annotated.
4. The method according to any one of claims 1 to 3, wherein the annotation paradigm further includes annotation template information about the data to be annotated and its corresponding annotations.
5. The method according to claim 1, further comprising: Obtain the modification instruction information input by the user in response to the annotation prompt information, and adjust the annotation prompt information according to the modification instruction information.
6. The method according to claim 5, further comprising: Input the annotation prompt information and the test data to be annotated into the second model to obtain the predicted annotation information corresponding to the test data output by the second model; Based on the predicted annotation information, obtain annotation evaluation information about the annotation prompt information.
7. The method according to claim 6, further comprising: The annotation prompt information is adjusted based on the annotation evaluation information.
8. The method according to claim 7, wherein adjusting the annotation prompt information based on the annotation evaluation information includes: The annotation evaluation information is input into the first model to obtain the adjusted annotation prompt information output by the first model based on the adjusted annotation paradigm.
9. The method according to claim 7, wherein adjusting the annotation prompt information based on the annotation evaluation information includes: In response to the user adjusting the annotation paradigm based on the annotation evaluation information, the adjusted annotation paradigm is re-input into the first model to obtain the adjusted annotation prompt information output by the first model based on the adjusted annotation paradigm.
10. The method according to claim 1, wherein the code information includes first code information for inputting the annotation prompt information and the data to be annotated into a third model and obtaining the annotation information output by the third model.
11. The method according to claim 10, wherein the code information further includes second code information for performing target processing on the data to be labeled and / or the labeling information, the target processing including at least one of data format cleaning, data table processing, and anomaly detection processing.
12. The method according to claim 10, wherein the code information further includes third code information for retrieving the annotation information and determining the authenticity of the annotation information based on the retrieval results.
13. The method of claim 12, further comprising: Obtain multiple annotation information that meet the preset conditions for authenticity, and adjust the annotation prompt information according to the multiple annotation information and their corresponding data to be annotated; This enables the first model to output the latest annotation service corresponding to the adjusted annotation prompt information.
14. An apparatus for automatic labeling, comprising: The annotation prompt information module is used to obtain the annotation paradigm of the target annotation task input by the user, input the annotation paradigm into the first model, and obtain the annotation prompt information output by the first model. The annotation paradigm includes the case data to be annotated and the annotation task description information. The annotation service module is used to respond to the intelligent capability generation request initiated by the user in response to the annotation prompt information, so that the first model outputs the annotation service corresponding to the annotation prompt information. The annotation service is used to automatically annotate the data to be annotated. The annotation service includes the annotation prompt information and code information for providing automatic annotation capabilities for the target annotation task based on the annotation prompt information.
15. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.
16. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method as claimed in any one of claims 1 to 13.
17. A computer program product having at least one instruction stored thereon, characterized in that, When the at least one instruction is executed by the processor, it implements the steps of the method according to any one of claims 1 to 13.