Method, device, storage medium and electronic device for labeling

Abstract

The embodiment of the specification discloses a kind of for marking method, apparatus, storage medium and electronic equipment, first obtain the marking result corresponding to one or more marking questions corresponding to marking task;After the automatic checking service is carried out to the marking result, the corresponding inspection result is obtained;Then according to the inspection result and pre-configured decision condition, corresponding operation is executed for the marking result.

Description

Technical Field

[0001] This invention relates to computer technology, and more particularly to a method, apparatus, storage medium, and electronic device for annotation. Background Technology

[0002] In recent years, the AI ​​(Artificial Intelligence) industry has developed rapidly. With the diversification of business, model training constantly pursues higher data quality, resulting in labeling tasks that are continuously changing, diverse, and highly challenging. However, regardless of the changes in tasks, the goal is to achieve faster delivery. But "fast production" does not equal "fast delivery." "AI + efficiency" in the single-point labeling process is only one of the influencing factors. Many other factors may prolong delivery time, such as insufficient production capacity and re-labeling due to labeling errors. Summary of the Invention

[0003] The purpose of the embodiments in this specification is to provide a method, apparatus, storage medium, and electronic device for labeling.

[0004] This specification provides a method for annotation, addressing the pain points of incomplete intelligent link coverage and lack of service mechanisms. It proposes deploying an AI-powered automatic inspection service in the annotation task inspection phase to achieve automatic inspection of annotation results. Through real-time AI, potential problems can be detected promptly, shortening the delivery cycle of annotation tasks and improving quality. By empowering the inspection phase with AI, the intelligent link is streamlined, and a closed-loop optimization system is established, significantly improving the delivery efficiency and quality of annotation tasks. The method includes: Obtain the annotation results for one or more annotation titles corresponding to the annotation task; The annotation results are automatically checked using the automatic checking service to obtain the corresponding check results; Based on the inspection results and pre-configured decision conditions, perform corresponding operations on the annotation results.

[0005] Furthermore, the annotation results include manually annotated results; The step of obtaining the annotation results corresponding to one or more annotation topics for the annotation task includes: In response to users annotating one or more annotation topics corresponding to the annotation task, the corresponding manual annotation results are obtained.

[0006] Further, the step of automatically checking the annotation results according to the automatic checking service to obtain the corresponding check results includes: The manual annotation results are verified using the first auxiliary verification service. If the verification passes, the manual annotation results are automatically checked using the automatic checking service to obtain the corresponding check results.

[0007] Further, the step of performing auxiliary verification on the manually labeled results according to the first auxiliary verification service includes: In response to the annotation submission trigger event corresponding to the annotation task, the manual annotation result is assisted in verification according to the first auxiliary verification service.

[0008] Furthermore, if the verification passes, the automatic checking service is used to automatically check the manually labeled results to obtain the corresponding check results, including: If a corresponding verification result is obtained within a preset time range after the annotation submission trigger event and the verification result indicates that the verification has passed, the manual annotation result is automatically checked according to the automatic checking service to obtain the corresponding check result.

[0009] Furthermore, the method also includes: If no corresponding verification result is obtained within a preset time range after the annotation submission trigger event occurs, the manual annotation result is automatically checked according to the automatic checking service to obtain the corresponding check result.

[0010] Furthermore, the inspection results include the degree of error corresponding to the annotation results; The step of performing corresponding operations on the annotation results based on the inspection results and pre-configured decision conditions includes: Based on pre-configured decision conditions, the operation type corresponding to the error level of the annotation result is obtained, and the corresponding operation is performed on the annotation result according to the operation type.

[0011] Furthermore, the degree of error includes correctness, and the operation type it maps to includes acceptance; The step of performing the corresponding operation on the annotation result according to the operation type includes: Generate an acceptance report corresponding to the annotation results.

[0012] Furthermore, the error level includes doubtful, and the mapped operation type includes quality inspection; The step of performing the corresponding operation on the annotation result according to the operation type includes: Obtain the manual inspection results corresponding to the labeled results.

[0013] Furthermore, obtaining the manual inspection result corresponding to the annotation result also includes: The manual inspection results are verified using the second auxiliary verification service. If the verification passes and the manual inspection result indicates that the inspection has passed, an acceptance report corresponding to the annotation result is generated.

[0014] Furthermore, the error level includes errors, and the operation type it maps to includes annotations; The step of performing the corresponding operation on the annotation result according to the operation type includes: Obtain the latest annotation results corresponding to the labeled title.

[0015] Furthermore, the response to the user annotating one or more annotation topics corresponding to the annotation task and obtaining the corresponding manual annotation results includes: Based on the pre-annotation service, pre-annotation information corresponding to one or more annotation topics for the annotation task is generated and presented to the user. In response to the user annotating the title based on the pre-annotation information, the corresponding manual annotation results are obtained.

[0016] Furthermore, the response to the user annotating one or more annotation topics corresponding to the annotation task and obtaining the corresponding manual annotation results includes: Based on the auxiliary annotation service, generate auxiliary annotation information for one or more annotation titles corresponding to the annotation task, and present the auxiliary annotation information to the user; In response to the user annotating the title based on the auxiliary annotation information, the corresponding manual annotation result is obtained.

[0017] Furthermore, the method also includes: The system presents users with service descriptions and service activation steps for each of the multiple services. The service activation steps include at least one of automatic checking, assisted labeling, pre-labeling, and assisted verification. Obtain the automatic check service selected by the user from the plurality of services.

[0018] Furthermore, the presentation of service description information and service activation steps corresponding to multiple services to the user includes: The system presents users with service descriptions, service activation steps, and service capability application effects for multiple services. The service capability application effects include at least one of service efficiency, service quality, and service time.

[0019] Furthermore, the method also includes: For the same annotation task, a service capability comparison test was conducted by constructing a control group without the service configured and an experimental group with the service configured, to obtain the application effect of the service capability corresponding to the service.

[0020] This specification also provides an apparatus for labeling, comprising: The annotation module is used to obtain the annotation results for one or more annotation topics corresponding to an annotation task; The inspection module is used to automatically inspect the annotation results according to the automatic inspection service and obtain the corresponding inspection results; The decision module is used to perform corresponding operations on the annotation results based on the inspection results and pre-configured decision conditions.

[0021] This specification also provides a storage medium storing a computer program adapted to be loaded by a processor and to execute the steps of the method described above.

[0022] This specification also provides an electronic device, including a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method described above.

[0023] This specification also provides a computer program product having at least one instruction stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0024] According to the embodiments of this specification, in order to address the pain points of incomplete intelligent link coverage and lack of service mechanisms, it is proposed to deploy AI automatic inspection service in the inspection stage of annotation tasks to realize the automatic inspection function of annotation results. Through real-time AI, potential problems can be discovered in a timely manner, shortening the delivery cycle of annotation tasks and improving quality. By empowering the inspection stage with AI, the intelligent link can be opened up and a closed-loop optimization system can be established, which can significantly improve the delivery efficiency and quality of annotation tasks. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a method for annotation provided in an embodiment of this specification; Figure 2 A schematic diagram of a system framework for annotation provided in the embodiments of this specification; Figure 3 A schematic diagram of a device for labeling provided in an embodiment of this specification; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0027] Please see Figure 1 This is a flowchart illustrating a method for annotation provided in an embodiment of this specification. In this embodiment, the annotation method is applied to an annotation apparatus (hereinafter referred to as a "annotation apparatus") or an electronic device equipped with an annotation apparatus as described in this embodiment. The following will focus on... Figure 1 The process shown will be described in detail. The annotation method may specifically include the following steps: S102, obtain the annotation results corresponding to one or more annotation topics for the annotation task.

[0028] In some embodiments, a labeling task refers to labeling raw unstructured data (images, text, speech, video, etc.) with structured semantic labels, ultimately generating a high-quality dataset that can be directly used for supervised learning, model training, and evaluation. In some embodiments, a labeling title refers to the smallest independent labeling unit after the labeling task is broken down; it is the most basic execution unit of the labeling work. Labeling titles include, but are not limited to, the original data to be labeled, labeling rules, and labeling regions; this example embodiment does not impose any special limitations on these. In some embodiments, the labeling result is the final submitted label or marker content after strictly following the labeling rules and / or labeling regions to label the original data for each labeling title; it is the final output of the labeling title. In some embodiments, the labeling result can be a manually labeled result obtained by manually labeling one or more labeling titles corresponding to the labeling task, or it can be an automatically labeled result obtained by a large model automatically labeling one or more labeling titles corresponding to the labeling task.

[0029] S104, The annotation results are automatically checked according to the automatic inspection service to obtain the corresponding inspection results.

[0030] In some embodiments, the automatic inspection service is an AI service used to automatically inspect annotation results. The AI ​​service refers to artificial intelligence capabilities encapsulated into standardized, callable, and reusable online services / interfaces, providing AI capability calls to the outside world via the network (API / SDK). This intelligent capability service eliminates the need for users to perform underlying modeling, training, and deployment, allowing for direct on-demand use. In some embodiments, after submitting annotation results, the automatic inspection service automatically inspects the results to obtain corresponding inspection results. This achieves quality control of the annotation task through the automatic inspection service, enabling more timely and comprehensive problem detection using AI. The inspection results are not limited to "passed" or "failed," and this example embodiment does not impose any specific limitations on this.

[0031] S106, Based on the inspection results and pre-configured decision conditions, perform corresponding operations on the annotation results.

[0032] In some embodiments, pre-configured decision conditions include mapping relationships between various inspection results and various operation types. For example, "inspection passed" maps to the operation type of acceptance, and "inspection failed" maps to the operation type of annotation. For instance, if the inspection result is "inspection passed," a corresponding acceptance report is automatically generated based on the annotation results and / or annotation titles. This allows business stakeholders to evaluate the annotation task from a data aggregation perspective (e.g., annotation speed, annotation quality, etc.) by reviewing the acceptance report, thereby determining whether the annotation task meets expectations. The acceptance report includes data distribution of the annotation results, quality information such as inspection accuracy, and data details of the annotation results, i.e., points that require special attention in the annotation results. As another example, if the inspection result is "inspection failed," the annotation results are discarded, and the annotation titles are re-annotated manually or automatically to obtain the latest annotation results corresponding to the annotation titles.

[0033] According to the embodiments of this specification, in order to address the pain points of incomplete intelligent link coverage and lack of service mechanisms, it is proposed to deploy AI automatic inspection service in the inspection stage of annotation tasks to realize the automatic inspection function of annotation results. Through real-time AI, potential problems can be discovered in a timely manner, shortening the delivery cycle of annotation tasks and improving quality. By empowering the inspection stage with AI, the intelligent link can be opened up and a closed-loop optimization system can be established, which can significantly improve the delivery efficiency and quality of annotation tasks.

[0034] In some embodiments, the annotation results include manual annotation results; wherein, obtaining the annotation results corresponding to one or more annotation topics corresponding to the annotation task includes: in response to the user annotating one or more annotation topics corresponding to the annotation task, obtaining the corresponding manual annotation results.

[0035] In some embodiments, the automatic checking of the annotation results using an automatic checking service to obtain corresponding check results includes: performing auxiliary verification on the manual annotation results using a first auxiliary verification service; if the verification passes, automatically checking the manual annotation results using the automatic checking service to obtain corresponding check results. In some embodiments, the first auxiliary verification service is an AI service for auxiliary verification of manual annotation results. Here, auxiliary verification refers to verifying whether the manual annotation results contain errors, such as verifying whether there are clearly defined errors or high-frequency, low-level errors in the manual annotation results. The first auxiliary verification service needs to pre-configure conditional rules; for example, a conditional rule could be that Arabic numerals cannot appear in the annotation results. In some embodiments, if the first auxiliary verification fails, a warning or block will be issued for the manual annotation results. Only if the first auxiliary verification passes will the automatic checking of the manual annotation results using the automatic checking service be triggered.

[0036] In some embodiments, the step of performing auxiliary verification on the manual annotation result according to the first auxiliary verification service includes: responding to the annotation submission trigger event corresponding to the annotation task, and performing auxiliary verification on the manual annotation result according to the first auxiliary verification service. In some embodiments, in response to the user submitting the manual annotation result, auxiliary verification of the manual annotation result through the first auxiliary verification service is automatically triggered.

[0037] In some embodiments, the step of automatically checking the manual annotation result according to the automatic checking service to obtain the corresponding check result if the verification passes includes: if a corresponding verification result is obtained within a preset time range after the annotation submission trigger event and the verification result indicates that the verification passes, the manual annotation result is automatically checked according to the automatic checking service to obtain the corresponding check result. In some embodiments, a timer starts after the user submits the manual annotation result. If a verification result corresponding to the first auxiliary verification service is obtained within a preset time range (e.g., 20 seconds) and the verification result indicates that the auxiliary verification passes, the auxiliary verification of the manual annotation result will be automatically triggered through the first auxiliary verification service.

[0038] In some embodiments, the method further includes: if no corresponding verification result is obtained within a preset time range after the annotation submission trigger event occurs, automatically checking the manual annotation result according to the automatic checking service to obtain the corresponding check result. In some embodiments, a timer starts after the user submits the manual annotation result, and if no verification result corresponding to the first auxiliary verification service is obtained within a preset time range (e.g., 20 seconds), then the auxiliary verification of the manual annotation result is directly triggered through the first auxiliary verification service.

[0039] In some embodiments, the inspection result includes the error level corresponding to the annotation result; wherein, the step of performing corresponding operations on the annotation result based on the inspection result and pre-configured decision conditions includes: obtaining the operation type mapped to the error level corresponding to the annotation result based on the pre-configured decision conditions, and performing corresponding operations on the annotation result based on the operation type. In some embodiments, the inspection result includes the error level corresponding to the annotation result, and the error level includes, but is not limited to, correct, questionable, and incorrect, etc., which are not specifically limited in this example embodiment. In some embodiments, the pre-configured decision conditions include the mapping relationship between multiple different error levels and multiple operation types. For example, the operation type mapped to "correct" is acceptance, the operation type mapped to "incorrect" is annotation, and the operation type mapped to "questionable" is quality inspection. In some embodiments, the operation corresponding to the operation type mapped to the error level is performed on the annotation result.

[0040] In some embodiments, the error level includes "correct," and the mapped operation type includes "acceptance." The step of performing the corresponding operation on the annotation result according to the operation type includes generating an acceptance report corresponding to the annotation result. In some embodiments, if the error level is "correct" and the operation type mapped to that error level is "acceptance," a corresponding acceptance report will be automatically generated based on the annotation result and / or annotation title. This allows the business party to perform a multi-dimensional evaluation of the annotation task from a data aggregation perspective by viewing the acceptance report, thereby determining whether the annotation task meets expectations.

[0041] In some embodiments, the error level includes doubtful, and the mapped operation type includes quality inspection; The step of performing corresponding operations on the annotation results according to the operation type includes: obtaining the manual inspection result corresponding to the annotation result. In some embodiments, if the error level is questionable and the operation type mapped to the error level is quality inspection, the annotation result will be manually inspected to obtain the corresponding manual inspection result. That is, the manual inspection determines whether there is a problem with the annotation result. If the manual inspection result is that there is no problem, the corresponding acceptance report will be automatically generated based on the annotation result and / or the annotation title. If the manual inspection result is that there is a problem, the manual inspection adjusts the annotation result based on the annotation result to obtain the adjusted annotation result. That is, the quality inspection result will also include the adjusted annotation result. Then, the corresponding acceptance report will be automatically generated based on the adjusted annotation result and / or the annotation title.

[0042] In some embodiments, obtaining the manual inspection result corresponding to the annotation result further includes: performing auxiliary verification on the manual inspection result according to the second auxiliary verification service; if the verification passes and the manual inspection result indicates that the inspection has passed, generating an acceptance report corresponding to the annotation result. In some embodiments, the second auxiliary verification service is an AI service for assisting in the verification of the manual inspection result. Here, auxiliary verification refers to verifying whether there are errors in the adjusted annotation result obtained after the manual adjustment of the annotation result, based on the manual determination that there are problems with the annotation result. For example, verifying whether there are clearly defined errors or high-frequency low-level errors in the adjusted annotation result. The second auxiliary verification service needs to pre-configure conditional rules. For example, the conditional rule can be that Arabic numerals cannot appear in the annotation result. In some embodiments, if the second auxiliary verification fails, the adjusted annotation result will be warned or blocked. Only if the second auxiliary verification passes will the corresponding acceptance report be automatically generated based on the adjusted annotation result and / or the annotation title.

[0043] In some embodiments, the error level includes error, and the operation type it maps to includes annotation; wherein, performing the corresponding operation on the annotation result according to the operation type includes: obtaining the latest annotation result corresponding to the annotation title. In some embodiments, if the error level is error and the operation type mapped to the error level is annotation, the annotation result will be discarded, and the latest annotation result corresponding to the annotation title will be obtained again by manually or automatically annotating the annotation title.

[0044] In some embodiments, the step of responding to a user annotating one or more annotation topics corresponding to an annotation task and obtaining corresponding manual annotation results includes: generating pre-annotation information corresponding to one or more annotation topics corresponding to the annotation task based on a pre-annotation service, and presenting the pre-annotation information to the user; responding to a user annotating the annotation topics based on the pre-annotation information and obtaining corresponding manual annotation results. In some embodiments, the pre-annotation service is an AI service used to pre-annotate before manual annotation. Pre-annotation refers to automatically generating initial annotation results (i.e., pre-annotation information) on the original data to be annotated using a trained large model, serving as a draft or preliminary version for manual annotation, rather than directly serving as the final annotation result. In some embodiments, by generating pre-annotation information corresponding to annotation topics through a pre-annotation service and presenting the pre-annotation information to the user, the user can manually annotate the annotation topics based on the pre-annotation information and obtain corresponding manual annotation results.

[0045] In some embodiments, the step of responding to a user annotating one or more annotation topics corresponding to an annotation task and obtaining corresponding manual annotation results includes: generating auxiliary annotation information corresponding to one or more annotation topics corresponding to the annotation task based on an auxiliary annotation service, and presenting the auxiliary annotation information to the user; responding to a user annotating the annotation topics based on the auxiliary annotation information and obtaining corresponding manual annotation results. In some embodiments, the auxiliary annotation service is an AI service used to assist in annotation before manual annotation. Assisted annotation refers to at least one of the following (i.e., auxiliary annotation information) provided by a trained large model: annotation suggestions, completion, verification, error correction, and recommended labels / candidate results, to help annotators reduce repetitive work and improve annotation efficiency. In some embodiments, during the user's manual annotation process for annotation topics, auxiliary annotation information corresponding to the annotation topic is generated through the auxiliary annotation service, and the user is presented with the auxiliary annotation information so that the user can manually annotate the annotation topic based on the auxiliary annotation information and obtain corresponding manual annotation results.

[0046] In some embodiments, the method further includes: presenting the user with service description information and service activation stages corresponding to multiple services, wherein the service activation stage includes at least one of automatic checking, assisted annotation, pre-annotation, and assisted verification; and obtaining the automatic checking service selected by the user from the multiple services. In some embodiments, the service types of the multiple services (i.e., AI services) include, but are not limited to, automatic checking services, first assisted verification services, second assisted verification services, pre-annotation services, and assisted annotation services, etc., and this example embodiment does not make any special limitations on this. In some embodiments, by constructing an online service center, users can view the service description information and service activation stages (i.e., the stages of service action) of different services, thereby assisting users in filtering services and obtaining the automatic checking service selected by the user from multiple services, wherein the service activation stage includes at least one of automatic checking, assisted annotation, pre-annotation, and assisted verification, and the service description information includes, but is not limited to, service name, service type (e.g., text, image, etc.), service usage scenario (e.g., intent recognition, text rewriting, image classification, format parsing, text summarization, key information extraction, content question answering, relevance judgment, etc.), application field (general, medical, financial, etc.), etc., and this example embodiment does not make any special limitations on this.

[0047] In some embodiments, presenting users with service descriptions and service activation stages corresponding to multiple services includes: presenting users with service descriptions, service activation stages, and service capability application effects for each service. The service capability application effects include at least one of service efficiency (e.g., annotation efficiency, inspection efficiency), service quality (e.g., annotation accuracy, inspection accuracy), and service time (e.g., annotation time, inspection time). In some embodiments, the online service center also presents users with the service capability application effects of each service, allowing users to view the application effects of different services to assist in service selection.

[0048] In some embodiments, the method further includes: for the same annotation task, conducting a service capability comparison test by constructing a control group without the service configured and an experimental group with the service configured, to obtain the service capability application effect corresponding to the service. In some embodiments, the service capability comparison test refers to testing the service capability application effect of a service through a comparative experiment (A / B experiment). An A / B experiment is a data-driven testing method that randomly assigns users to different groups (such as group A and group B) and compares key indicators to scientifically quantify the differences in product or service effectiveness. In some embodiments, the A / B experiment objectively measures the service capability application effect, constructs a data-driven capability operation system, and creates a configuration distribution ratio through annotation tasks. This allows for a service capability comparison test for the same annotation task by constructing a control group without the service configured and an experimental group with the service configured. By focusing on the three stages of labeling, testing, and verification, the method obtains the service capability application effect corresponding to the service, ensuring that the results are scientifically reliable and can be used as a reference for subsequent deployment decisions.

[0049] Figure 2 This is a schematic diagram of a system framework for annotation provided in an embodiment of this specification.

[0050] like Figure 2As shown, in the application layer stage, the data layer performs light processing on the raw data to obtain labelable data. The annotation layer automatically annotates the labelable data to obtain automatic annotation results. Alternatively, the annotation layer manually annotates the labelable data based on auxiliary annotation (auxiliary annotation) or pre-annotation (pre-annotation) to obtain manual annotation results. The manual annotation results are then subject to auxiliary verification. The inspection layer automatically checks the annotation results (manual annotation results or manual annotation results that have passed auxiliary verification) and makes a decision based on the automatic inspection results. If the decision is rejected, it needs to be returned to the annotation layer for re-annotation. If the decision is approved, the acceptance layer automatically generates the corresponding acceptance report based on the annotation results. If the decision is questionable, the annotation results are manually checked, and the manual check results are then subject to auxiliary verification. If the auxiliary verification is passed, the acceptance layer automatically generates the corresponding acceptance report based on the annotation results. Finally, the acceptance report is manually reviewed to determine whether the annotation task has been accepted.

[0051] Figure 3 This is a schematic diagram of a labeling device provided in an embodiment of this specification. The labeling device (hereinafter referred to as "labeling device 1") can be implemented as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the labeling device 1 includes a labeling module 11, an inspection module 12, and a decision module 13.

[0052] Annotation module 11 is used to obtain annotation results for one or more annotation topics corresponding to the annotation task; Inspection module 12 is used to automatically inspect the annotation results according to the automatic inspection service and obtain the corresponding inspection results; The decision module 13 is used to perform corresponding operations on the annotation results based on the inspection results and pre-configured decision conditions.

[0053] In some embodiments, the annotation results include manual annotation results; wherein, obtaining the annotation results corresponding to one or more annotation topics corresponding to the annotation task includes: in response to the user annotating one or more annotation topics corresponding to the annotation task, obtaining the corresponding manual annotation results.

[0054] In some embodiments, the step of automatically checking the annotation results according to the automatic checking service to obtain the corresponding checking results includes: performing auxiliary verification on the manual annotation results according to the first auxiliary verification service; if the verification passes, automatically checking the manual annotation results according to the automatic checking service to obtain the corresponding checking results.

[0055] In some embodiments, the step of performing auxiliary verification on the manual annotation results according to the first auxiliary verification service includes: responding to the annotation submission trigger event corresponding to the annotation task, and performing auxiliary verification on the manual annotation results according to the first auxiliary verification service.

[0056] In some embodiments, if the verification passes, the step of automatically checking the manual annotation result according to the automatic checking service to obtain the corresponding check result includes: if the corresponding verification result is obtained within a preset time range after the annotation submission trigger event occurs and the verification result indicates that the verification passes, the manual annotation result is automatically checked according to the automatic checking service to obtain the corresponding check result.

[0057] In some embodiments, the annotation device 1 is further configured to: if no corresponding verification result is obtained within a preset time range after the annotation submission trigger event occurs, automatically check the manual annotation result according to the automatic checking service to obtain the corresponding check result.

[0058] In some embodiments, the inspection result includes the error level corresponding to the annotation result; wherein, the step of performing corresponding operations on the annotation result based on the inspection result and pre-configured decision conditions includes: obtaining the operation type mapped to the error level corresponding to the annotation result based on the pre-configured decision conditions, and performing corresponding operations on the annotation result based on the operation type.

[0059] In some embodiments, the error level includes correctness, and the operation type it maps to includes acceptance; wherein, performing the corresponding operation on the annotation result according to the operation type includes: generating an acceptance report corresponding to the annotation result.

[0060] In some embodiments, the error level includes doubtful, and the operation type it maps to includes quality inspection; wherein, performing the corresponding operation on the annotation result according to the operation type includes: obtaining the manual inspection result corresponding to the annotation result.

[0061] In some embodiments, obtaining the manual inspection result corresponding to the annotation result further includes: performing auxiliary verification on the manual inspection result according to the second auxiliary verification service; if the verification is passed and the manual inspection result indicates that the inspection is passed, generating an acceptance report corresponding to the annotation result.

[0062] In some embodiments, the error level includes error, and the operation type it maps to includes annotation; wherein, performing the corresponding operation on the annotation result according to the operation type includes: obtaining the latest annotation result corresponding to the annotation title.

[0063] In some embodiments, the step of responding to a user annotating one or more annotation topics corresponding to an annotation task and obtaining corresponding manual annotation results includes: generating pre-annotation information corresponding to one or more annotation topics corresponding to an annotation task based on a pre-annotation service, and presenting the pre-annotation information to the user; and responding to a user annotating the annotation topics based on the pre-annotation information and obtaining corresponding manual annotation results.

[0064] In some embodiments, the step of responding to a user annotating one or more annotation topics corresponding to an annotation task and obtaining corresponding manual annotation results includes: generating auxiliary annotation information corresponding to one or more annotation topics corresponding to an annotation task based on an auxiliary annotation service, and presenting the auxiliary annotation information to the user; and responding to a user annotating the annotation topics based on the auxiliary annotation information and obtaining corresponding manual annotation results.

[0065] In some embodiments, the labeling device 1 is further configured to: present service description information and service activation steps corresponding to multiple services to the user, wherein the service activation steps include at least one of automatic checking, assisted labeling, pre-labeling, and assisted verification; and obtain the automatic checking service selected by the user from the multiple services.

[0066] In some embodiments, presenting the user with service description information and service activation steps corresponding to multiple services includes: presenting the user with service description information, service activation steps, and service capability application effects corresponding to multiple services, wherein the service capability application effects include at least one of service efficiency, service quality, and service time.

[0067] In some embodiments, the annotation device 1 is further configured to: for the same annotation task, conduct a service capability comparison test by constructing a control group without the service configured and an experimental group with the service configured, and obtain the service capability application effect corresponding to the service.

[0068] The above-described apparatus embodiments correspond to the aforementioned method embodiments. For detailed descriptions, please refer to the description in the method embodiments section; further details will not be repeated here. The apparatus embodiments are derived from the corresponding method embodiments and have the same technical effects. For detailed descriptions, please refer to the corresponding method embodiments.

[0069] This specification also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in this specification.

[0070] This specification also provides a computer program product that stores at least one instruction, which is loaded by the processor and executes the method described in this specification embodiment.

[0071] This specification also provides an electronic device, including a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and execute the method described in the embodiments of this specification.

[0072] The embodiments in this specification also provide Figure 4 The diagram shows the structure of the electronic device. Figure 4 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the above method.

[0073] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0074] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0078] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0079] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0080] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0081] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for annotation, comprising: Obtain the annotation results for one or more annotation titles corresponding to the annotation task; The annotation results are automatically checked using the automatic checking service to obtain the corresponding check results; Based on the inspection results and pre-configured decision conditions, perform corresponding operations on the annotation results.

2. The method according to claim 1, wherein the annotation results include manually annotated results; in, The process of obtaining the annotation results corresponding to one or more annotation topics for an annotation task includes: In response to users annotating one or more annotation topics corresponding to the annotation task, the corresponding manual annotation results are obtained.

3. The method according to claim 2, wherein automatically checking the annotation results using an automatic checking service to obtain corresponding check results includes: The manual annotation results are verified using the first auxiliary verification service. If the verification passes, the manual annotation results are automatically checked using the automatic checking service to obtain the corresponding check results.

4. The method according to claim 3, wherein the auxiliary verification of the manually labeled result based on the first auxiliary verification service includes: In response to the annotation submission trigger event corresponding to the annotation task, the manual annotation result is assisted in verification according to the first auxiliary verification service.

5. The method according to claim 4, wherein if the verification passes, the step of automatically checking the manually labeled result using an automatic checking service to obtain the corresponding check result includes: If a corresponding verification result is obtained within a preset time range after the annotation submission trigger event and the verification result indicates that the verification has passed, the manual annotation result is automatically checked according to the automatic checking service to obtain the corresponding check result.

6. The method according to claim 5, further comprising: If no corresponding verification result is obtained within a preset time range after the annotation submission trigger event occurs, the manual annotation result is automatically checked according to the automatic checking service to obtain the corresponding check result.

7. The method according to claim 1, wherein the inspection result includes the degree of error corresponding to the annotation result; in, The step of performing corresponding operations on the annotation results based on the inspection results and pre-configured decision conditions includes: Based on pre-configured decision conditions, the operation type corresponding to the error level of the annotation result is obtained, and the corresponding operation is performed on the annotation result according to the operation type.

8. The method of claim 7, wherein the degree of error includes correctness, and the operation type mapped thereto includes acceptance; in, The step of performing corresponding operations on the annotation results according to the operation type includes: Generate an acceptance report corresponding to the annotation results.

9. The method according to claim 7, wherein the error level includes doubtful, and the mapped operation type includes quality inspection; in, The step of performing corresponding operations on the annotation results according to the operation type includes: Obtain the manual inspection results corresponding to the labeled results.

10. The method according to claim 9, wherein obtaining the manual inspection result corresponding to the annotation result further includes: The manual inspection results are verified using the second auxiliary verification service. If the verification passes and the manual inspection result indicates that the inspection has passed, an acceptance report corresponding to the annotation result is generated.

11. The method of claim 7, wherein the error level includes errors, and the operation type mapped includes annotation; in, The step of performing corresponding operations on the annotation results according to the operation type includes: Obtain the latest annotation results corresponding to the labeled title.

12. The method according to claim 2, wherein responding to the user annotating one or more annotation topics corresponding to the annotation task and obtaining the corresponding manual annotation results includes: Based on the pre-annotation service, pre-annotation information corresponding to one or more annotation topics for the annotation task is generated and presented to the user. In response to the user annotating the title based on the pre-annotation information, the corresponding manual annotation results are obtained.

13. The method according to claim 2, wherein responding to the user annotating one or more annotation topics corresponding to the annotation task and obtaining the corresponding manual annotation results includes: Based on the auxiliary annotation service, generate auxiliary annotation information for one or more annotation titles corresponding to the annotation task, and present the auxiliary annotation information to the user; In response to the user annotating the title based on the auxiliary annotation information, the corresponding manual annotation result is obtained.

14. The method according to claim 1, further comprising: The system presents users with service descriptions and service activation steps for each of the multiple services. The service activation steps include at least one of automatic checking, assisted labeling, pre-labeling, and assisted verification. Obtain the automatic check service selected by the user from the plurality of services.

15. The method according to claim 14, wherein presenting the user with service description information and service activation steps corresponding to multiple services respectively includes: The system presents users with service descriptions, service activation steps, and service capability application effects for multiple services. The service capability application effects include at least one of service efficiency, service quality, and service time.

16. The method of claim 15, further comprising: For the same annotation task, a service capability comparison test was conducted by constructing a control group without the service configured and an experimental group with the service configured, to obtain the application effect of the service capability corresponding to the service.

17. An apparatus for labeling, comprising: The annotation module is used to obtain the annotation results for one or more annotation topics corresponding to an annotation task; The inspection module is used to automatically inspect the annotation results according to the automatic inspection service and obtain the corresponding inspection results; The decision module is used to perform corresponding operations on the annotation results based on the inspection results and pre-configured decision conditions.

18. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 16.

19. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method as claimed in any one of claims 1 to 16.

20. A computer program product having at least one instruction stored thereon, characterized in that, When the at least one instruction is executed by the processor, it implements the steps of the method according to any one of claims 1 to 16.

Images