Labeling method and device for model evaluation, equipment, storage medium and program product

Abstract

The invention relates to the technical field of computers, and discloses a labeling method and device for model evaluation, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a labeling template and an evaluation strategy of a to-be-evaluated model, and binding the labeling template to the evaluation strategy; if an evaluation task corresponding to the evaluation strategy is triggered, an evaluation interface of the evaluation task is displayed, and the evaluation interface comprises at least one evaluation item; and in response to a labeling operation for the at least one evaluation item, generating evaluation labeling content corresponding to the evaluation task. By implementing the technical scheme of the invention, labeling specifications are unified, the evaluation conclusion of the to-be-evaluated model can be quickly determined according to the evaluation labeling content, and the model evaluation labeling efficiency is improved, so that the iteration efficiency of the model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a labeling method, device, equipment, storage medium and program product for model evaluation. Background Art

[0002] Evaluation annotation is an indispensable part of model effect evaluation. It requires evaluation technicians to be able to annotate specific evaluation content so that these evaluation annotation information can be efficiently shared and used to adjust the model. At present, Excel tables are mainly used to summarize information such as evaluation questions, evaluation results, standard answers, etc., and annotators need to summarize the annotation results and classify them according to the summary content. However, different annotators have different annotation methods for the same error type, which is difficult to unify. This makes the later summary take a lot of time, resulting in low model evaluation efficiency, which affects the iteration efficiency of the model. Summary of the invention

[0003] In view of this, the present disclosure provides a model evaluation annotation method, device, equipment, storage medium and program product to solve the problem of low model evaluation efficiency.

[0004] In a first aspect, the present disclosure provides a labeling method for model evaluation, comprising: obtaining a labeling template and an evaluation strategy of a model to be evaluated, and binding the labeling template to the evaluation strategy; if an evaluation task corresponding to the evaluation strategy is triggered, displaying an evaluation interface of the evaluation task, the evaluation interface including at least one evaluation item; in response to a labeling operation on at least one evaluation item, generating evaluation labeling content corresponding to the evaluation task.

[0005] In a second aspect, the present disclosure provides a labeling device for model evaluation, comprising: an acquisition module, used to obtain the labeling template and evaluation strategy of the model to be evaluated, and bind the labeling template to the evaluation strategy; an evaluation display module, used to display the evaluation interface of the evaluation task when the evaluation task corresponding to the evaluation strategy is triggered, and the evaluation interface includes at least one evaluation item; a labeling module, used to generate evaluation labeling content corresponding to the evaluation task in response to a labeling operation on at least one evaluation item.

[0006] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the model evaluation labeling method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0007] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the model evaluation labeling method of the first aspect or any corresponding embodiment thereof.

[0008] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, which are used to enable a computer to execute the model evaluation annotation method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0009] The annotation method, device, equipment, storage medium and program product for model evaluation provided by the embodiments of the present disclosure obtain the annotation template and evaluation strategy of the model to be evaluated, bind the evaluation strategy with the annotation template, and display the corresponding evaluation interface in combination with the annotation template in the evaluation strategy when executing the evaluation task, so that the evaluation technician can evaluate and annotate the model according to the evaluation items of the annotation template displayed on the evaluation interface, thereby realizing standardized annotation for the model to be evaluated, avoiding the evaluation technicians from establishing their own standards and making it difficult to unify the evaluation results. Through the unified annotation specification, it is convenient to quickly determine the evaluation conclusion of the model to be evaluated according to the evaluation annotation content, improve the efficiency of model evaluation annotation, and thus improve the iteration efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 is a flowchart of a labeling method for model evaluation according to an embodiment of the present disclosure;

[0012] Figure 2 is a flowchart of another model evaluation annotation method according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic diagram of an interface of a marking template according to an embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram of an interface of an evaluation strategy according to an embodiment of the present disclosure;

[0015] Figure 5 is a flowchart of another model evaluation annotation method according to an embodiment of the present disclosure;

[0016] Figure 6is a schematic diagram of an evaluation interface according to an embodiment of the present disclosure;

[0017] Figure 7 is a schematic diagram of a specific process of a labeling method for model evaluation according to an embodiment of the present disclosure;

[0018] Figure 8 is a structural block diagram of a labeling device for model evaluation according to an embodiment of the present disclosure;

[0019] Fig. 9 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0021] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0022] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0023] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0025] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0026] At present, Excel tables are mainly used to summarize information such as evaluation questions, evaluation results, and standard answers, with each row serving as evaluation information for a question. If there is a problem with the evaluation question, or the result differs from the standard answer in detail, a column is added after the current row of the evaluation question as a record of the annotation results. After a round of evaluation is completed, the annotators need to summarize the annotation result columns and classify them according to the content. At the same time, different evaluation technicians have different and varied annotation methods for the same error type, which cannot be unified. This brings great pressure and labor costs to the later summary, resulting in a longer time-consuming later summary and low model evaluation efficiency, which in turn affects the iteration efficiency of the model.

[0027] Based on this, the core of the technical solution disclosed in the present invention is to set a labeling template and bind the labeling template with the evaluation strategy, thereby realizing standardized labeling for the model evaluation process, avoiding the evaluation technicians establishing their own standards, which makes it difficult to unify the evaluation labeling conclusions, thereby improving the model evaluation efficiency and iteration efficiency.

[0028] According to an embodiment of the present disclosure, an embodiment of a labeling method for model evaluation is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] In this embodiment, a model evaluation annotation method is provided, which can be used for computer devices, such as computers, mobile phones, tablet computers, etc. Figure 1 is a flow chart of a labeling method for model evaluation according to an embodiment of the present disclosure, such as Figure 1 As shown, the process includes the following steps:

[0030] Step S101, obtaining the annotation template and the evaluation strategy of the model to be evaluated, and binding the annotation template to the evaluation strategy.

[0031] The model to be evaluated is the model that needs to be evaluated. Specifically, the evaluation annotation of the model refers to marking or annotating the model data so as to evaluate the performance of the model. Through accurate evaluation annotation information, clear goals and standards can be provided for the model so that it can learn the correct patterns and features.

[0032] The annotation template is a specification or format for annotating model data to standardize the annotation content of the evaluation task. That is, the marking specification of the model data is specified in the annotation template. Specifically, the annotator can set the annotation template items according to the evaluation focus of the model to be evaluated, and generate the annotation template using the set annotation template items. Among them, the evaluation focus that the model to be evaluated needs to pay attention to can be obtained from the model development document, can also be obtained from the model performance related documents, or can be set by the data annotation team. The method of obtaining the evaluation focus is not specifically limited here.

[0033] The evaluation strategy is used to evaluate the performance of the model. It requires comparing the output of the sampled model with the actual results and measuring the model's capabilities based on the comparison results. The evaluation strategy is determined by the evaluator based on the evaluation requirements. The evaluation strategy includes various model indicators and specific evaluation methods for the model. Model indicators include accuracy, recall, F1 value, etc.

[0034] Associate the annotation template with the evaluation strategy to ensure that unified annotation data and methods can be used when evaluating and annotating the model, so as to avoid the impact of inconsistent annotation data and methods on the model evaluation results. Specifically, the annotation template and evaluation strategy used can be specified in the evaluation tool to associate the annotation template with the evaluation strategy; the annotation template and the evaluation strategy can also be integrated in the code corresponding to the evaluation strategy; the annotation data can also be processed according to the annotation template, and the corresponding evaluation strategy can be used for model evaluation. Of course, the evaluation strategy and the annotation template can also be bound by other binding methods, which are not specifically limited here, as long as the corresponding annotation template can be specified in the evaluation strategy. By binding the annotation template to the evaluation strategy, the accuracy and reliability of the evaluation results of the model to be evaluated can be ensured, and it also helps to objectively compare and analyze the model to be evaluated.

[0035] Step S102: if the evaluation task corresponding to the evaluation strategy is triggered, an evaluation interface of the evaluation task is displayed, and the evaluation interface includes at least one evaluation item.

[0036] Evaluation items are evaluation indicators set for the key evaluation points of the model to be evaluated, such as model capability error classification, block, error type, etc. A model evaluation system is deployed in the computer device. When the model evaluation system is triggered to perform an evaluation task for the model to be evaluated, the evaluation strategy corresponding to the model to be evaluated is called, and the corresponding evaluation interface is displayed according to the annotation template specified in the evaluation strategy.

[0037] The evaluation interface displays one or more evaluation items corresponding to the evaluation task, which can be displayed in the form of lists, tables, charts, etc., so that users can intuitively understand the various indicators of the evaluation task. Of course, the evaluation interface can also provide functional controls, such as start evaluation, pause evaluation, re-evaluation, view evaluation history, etc., to facilitate users to manage and control evaluation tasks.

[0038] Step S103: generating evaluation annotation content corresponding to the evaluation task in response to the annotation operation on at least one evaluation item.

[0039] The annotator annotates each evaluation item in the evaluation interface according to the running results of the model to be evaluated to generate the annotation results of each evaluation item. Accordingly, the evaluation interface can respond to the annotation operation of the annotator on the evaluation item, and parse the annotation operation to determine the annotation result of the annotator on the evaluation item, and generate the evaluation annotation content corresponding to the evaluation task according to the annotation result. At the same time, the evaluation annotation content can be stored so that the ability performance of the model to be evaluated can be determined according to the evaluation annotation content in the future, and when an error occurs, the error can be located according to the evaluation annotation content.

[0040] The annotation method for model evaluation provided in this embodiment obtains the annotation template and evaluation strategy of the model to be evaluated, binds the evaluation strategy with the annotation template, and displays the corresponding evaluation interface in combination with the annotation template in the evaluation strategy when executing the evaluation task, so that the evaluation technician can evaluate and annotate the model according to the evaluation items of the annotation template displayed on the evaluation interface, thereby realizing standardized annotation for the model to be evaluated, avoiding the evaluation technicians from establishing their own standards and making it difficult to unify the evaluation results. Through the unified annotation specification, it is convenient to quickly determine the evaluation conclusion of the model to be evaluated according to the evaluation annotation content, improve the efficiency of model evaluation annotation, and thus improve the iteration efficiency of the model.

[0041] In this embodiment, a model evaluation annotation method is provided, which can be used for computer devices, such as computers, mobile phones, tablet computers, etc. Figure 2 is a flow chart of a labeling method for model evaluation according to an embodiment of the present disclosure, such as Figure 2 As shown, the process includes the following steps:

[0042] Step S201, obtaining the annotation template and the evaluation strategy of the model to be evaluated, and binding the annotation template to the evaluation strategy.

[0043] Specifically, the above step S201 includes:

[0044] Step S2021, obtaining evaluation requirement information of the model to be evaluated.

[0045] The evaluation requirement information is the actual business requirement for the model to be evaluated. Specifically, the model to be evaluated is developed based on the actual application scenario, and the evaluation requirement information can be determined based on the actual application scenario; the development of the model to be evaluated has corresponding development documents, and the evaluation requirement information can also be extracted from the development documents of the model to be evaluated.

[0046] In a specific example, a model evaluation system is deployed in a computer device. Before starting the evaluation task for the model to be evaluated, the evaluation technicians participate in meetings or discussions based on the development documents and actual application scenarios of the model to be evaluated to determine the evaluation focus (e.g., this evaluation only needs to mark the error distribution, the amount of the business line, etc.) and a unified evaluation caliber (e.g., -1 point for poor performance, 2 points for good performance, and no other scores, etc.). By unifying the evaluation caliber, different labeling specifications can be established for different performances of the model.

[0047] Determine the evaluation requirement information corresponding to the model to be evaluated based on the meeting or discussion results, and upload the evaluation requirement information to the evaluation system in the computer device so that the evaluation system can obtain the corresponding evaluation requirement information before starting the evaluation task.

[0048] Step S2022, in response to the template item creation operation based on the evaluation requirement information, generating a marking template according to the template item creation operation.

[0049] The model evaluation system has a configuration interface for template items. The evaluation technicians create template items in the template item configuration interface provided by the model evaluation system according to the evaluation requirement information. Accordingly, the model evaluation system can respond to the template item creation operation of the evaluation technicians, generate multiple template items, and use multiple template items to build a labeling template. Among them, template items support multiple types such as numbers, strings, options, classifiers, and text.

[0050] like Figure 3 As shown, the model evaluation system has a label template management component. By clicking the label template management component, you can trigger the template item configuration for the label template, enter the management label item display interface, configure each template item in the management label item display interface, pop up the template item editing interface, edit the template item in the template item editing interface, and obtain the corresponding template item. After completing the editing of each template item, assemble each template item into a label template. The evaluation technician can name the label template so that he can directly locate the corresponding label template for use in the subsequent evaluation task through the naming information.

[0051] In some optional implementations, the method further includes: determining a marking range of the template item in response to a marking setting operation on the template item.

[0052] The annotation range is the numerical range allowed for annotating a template item with a numerical value. For one or more template items that can set a numerical range, the evaluation technician can set the annotation range of the template item when creating the template item. Accordingly, the model evaluation system can respond to the evaluation technician's annotation setting operation on the template item and obtain the annotation range corresponding to the template item. Specifically, each template item can have a preset annotation range. For example, the score of the model to be evaluated can be specified by the option type to only be selected from the annotation range of -1, 0, 1.

[0053] By setting the annotation range of the template item, the evaluation technicians can annotate the evaluation tasks according to the annotation range, thus achieving the unification of the annotation range, which is conducive to the subsequent rapid unification of the annotation results.

[0054] Step S2023, in response to the policy writing operation for the model to be evaluated, generating an evaluation strategy based on the policy writing operation.

[0055] The model evaluation system has an editing interface for evaluation strategies. Evaluation technicians can write evaluation strategies in the editing interface according to the application scenarios and evaluation directions of the models to be evaluated. Accordingly, the model evaluation system can respond to the strategy writing operations of the evaluation technicians in the editing interface and generate evaluation strategies for triggering evaluation tasks.

[0056] like Figure 4 As shown, the model evaluation system has the function of realizing automated logic. By clicking the automated logic component, the writing operation for the evaluation strategy is triggered, and the editing interface of the evaluation strategy is entered. The evaluation strategy is written in the editing interface to obtain the corresponding evaluation strategy.

[0057] Step S2024: In response to a specified operation on the annotation template in the evaluation strategy, the annotation template is bound to the evaluation strategy.

[0058] Different annotation templates have different identification information, that is, each annotation template has a corresponding unique identifier. In the editing interface of the evaluation strategy, the evaluation technician's specified operations on the annotation template are monitored through event listeners or callback functions. When the specified operation is detected, the annotation template to be bound is determined from the specified operation, the annotation template is associated with the evaluation strategy, and the evaluation strategy bound to the annotation template is saved. During the evaluation process, the evaluation strategy can be read in the relevant code logic, and the annotation template associated with it can be obtained, and then the corresponding evaluation interface can be displayed according to the annotation template.

[0059] Step S202: If the evaluation task corresponding to the evaluation strategy is triggered, the evaluation interface of the evaluation task is displayed, and the evaluation interface includes at least one evaluation item. Figure 1Step S101 of the illustrated embodiment will not be described in detail here.

[0060] Step S203: In response to the annotation operation for at least one evaluation item, generate evaluation annotation content corresponding to the evaluation task. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0061] The annotation method for model evaluation provided in this embodiment creates the required template items through the evaluation requirement information of the model to be evaluated, and generates the annotation template according to the template items, thereby realizing standardized annotation for the model to be evaluated through the annotation template, avoiding the impact of inconsistent annotation techniques on the subsequent summary of annotation results, and saving summary time and labor summary costs. By writing an evaluation strategy, a unified evaluation of the model to be evaluated is achieved, and the model evaluation is performed in the same evaluation method, thereby improving the evaluation consistency of the model. By specifying the annotation template in the evaluation strategy, the model evaluation is performed according to the annotation template when executing the evaluation task, and the unification of the evaluation method and the annotation method is achieved, which is conducive to improving the efficiency of model evaluation.

[0062] In this embodiment, a model evaluation annotation method is provided, which can be used for computer devices, such as computers, mobile phones, tablet computers, etc. Figure 5 is a flow chart of a labeling method for model evaluation according to an embodiment of the present disclosure, such as Figure 5 As shown, the process includes the following steps:

[0063] Step S301, obtain the annotation template and evaluation strategy of the model to be evaluated, and bind the annotation template to the evaluation strategy. Figure 2 Step S201 of the illustrated embodiment will not be described in detail here.

[0064] Step S302: if the evaluation task corresponding to the evaluation strategy is triggered, an evaluation interface of the evaluation task is displayed, and the evaluation interface includes at least one evaluation item.

[0065] Specifically, the evaluation items include one or more of the text evaluation items, link evaluation items, and overall evaluation items. Among them, the text evaluation items are detailed evaluation items for the model to be evaluated, and text annotation refers to marking the details of a certain evaluation content in an evaluation task to highlight the key content so as to accurately locate useful information in the annotation results.

[0066] Among them, link evaluation items are evaluation items for each module of the model to be evaluated. Link annotation refers to annotating a module of the model to be evaluated in an evaluation task. Module annotation does not care about content details. Unlike word annotation, module annotation focuses on a wider range and only gives overall suggestions for the block.

[0067] Among them, the overall evaluation item is the evaluation item for the entire model to be evaluated, and the overall annotation refers to the annotation for the entire evaluation task. It focuses on a wider range than the module annotation, such as the overall score of the current evaluation.

[0068] Evaluation strategies can provide text evaluation methods, execution link evaluation methods, and overall evaluation methods. If you are concerned about the details, you can use text annotation in the form of word marking; if you are in doubt about a certain link in the model execution link, you can use the link annotation in the form of modules; if you do not pay attention to the details and only need to give a judgment on the entire evaluation task, you can use the overall annotation. Different annotation methods can be customized in the written evaluation strategy.

[0069] Specifically, the above step S302 may include:

[0070] Step S3021: If the evaluation strategy represents the text evaluation method of the model to be evaluated, call the text annotation template corresponding to the text evaluation method, and display the text evaluation items corresponding to the text annotation template on the evaluation interface.

[0071] If the evaluation strategy represents the text evaluation method of the model to be evaluated, the evaluation strategy is parsed to determine the text annotation template bound to the evaluation strategy, and the corresponding text evaluation items are displayed on the evaluation interface according to the text annotation template, such as Figure 6 Correspondingly, the evaluation technician can mark the details of the model to be evaluated for the text evaluation item in the evaluation interface.

[0072] Step S3022: If the evaluation strategy represents the execution link evaluation method of the model to be evaluated, the link annotation template corresponding to the execution link evaluation method is called, and the link evaluation items corresponding to the link annotation template are displayed on the evaluation interface.

[0073] If the evaluation strategy represents the execution link evaluation method of the model to be evaluated, the evaluation strategy is parsed to determine the link annotation template bound to the evaluation strategy, and the corresponding link evaluation items are displayed on the evaluation interface according to the link annotation template, such as Figure 6 Accordingly, the evaluation technician can mark the execution link of the model to be evaluated according to the link evaluation item in the evaluation interface to determine whether the execution link of the model to be evaluated is correct.

[0074] Step S3023: if the evaluation strategy represents the overall evaluation method of the model to be evaluated, the overall annotation template corresponding to the overall evaluation method is called, and the overall evaluation items corresponding to the overall annotation template are displayed on the evaluation interface.

[0075] If the evaluation strategy represents the overall evaluation method of the model to be evaluated, the evaluation strategy is parsed to determine the overall annotation template bound to the evaluation strategy, and the corresponding overall evaluation items are displayed on the evaluation interface according to the overall annotation template, such as Figure 6 Accordingly, the evaluation technician can conduct an overall evaluation of the model to be evaluated based on the overall evaluation items in the evaluation interface to determine the overall capability performance of the model to be evaluated.

[0076] Step S303: generating evaluation annotation content corresponding to the evaluation task in response to the annotation operation on at least one evaluation item.

[0077] Specifically, the above step S303 includes:

[0078] Step S3031, in response to the annotation operation on the text evaluation item, annotate the text information of the model to be evaluated to generate a first annotation result.

[0079] For each text evaluation item displayed in the evaluation interface, the evaluation technician can mark the text information of the model to be evaluated according to the actual execution result of the model to be evaluated. Accordingly, the model evaluation system can parse the annotation operation of the text evaluation item to generate a first annotation result corresponding to the text evaluation item according to the annotation operation of the text evaluation item.

[0080] Step S3032, in response to the marking operation on the link evaluation item, each module of the model to be evaluated is evaluated to generate a second marking result.

[0081] For each link evaluation item displayed in the evaluation interface, the evaluation technician can evaluate and mark each module of the model to be evaluated according to the link execution result of the model to be evaluated. Accordingly, the model evaluation system can parse the marking operation of the link evaluation item to generate a second marking result corresponding to the link evaluation item according to the marking operation of the link evaluation item.

[0082] Step S3033, in response to the annotation operation on the overall evaluation item, the model to be evaluated is evaluated as a whole to generate a third annotation result.

[0083] For each overall evaluation item displayed in the evaluation interface, the evaluation technician can annotate the model to be evaluated as a whole according to the overall execution result of the model to be evaluated. Accordingly, the model evaluation system can parse the annotation operation of the overall evaluation item to generate a third annotation result corresponding to the overall evaluation item according to the annotation operation of the overall evaluation item.

[0084] Step S3034: determine the evaluation annotation content of the model to be evaluated based on the first annotation result, and / or the second annotation result, and / or the third annotation result.

[0085] The first annotation result, the second annotation result and the third annotation result correspond to different evaluation items, that is, each annotation result is used to characterize the importance of each evaluation item. If only one of the text evaluation item, link evaluation item and overall evaluation item is annotated, the evaluation annotation content is determined according to the annotation result of the corresponding evaluation item.

[0086] If multiple items in the text evaluation items, link evaluation items, and overall evaluation items are annotated, the importance of different evaluation items can be calculated, and corresponding weights can be assigned to each annotation result. The weighted average of each annotation result is used to determine the comprehensive annotation result of the model to be evaluated, and the evaluation annotation content of the model to be evaluated is determined based on the comprehensive annotation result. At the same time, consistency verification and correction can be performed on each annotation result to improve the accuracy and reliability of the evaluation annotation content.

[0087] In some optional implementations, the above method further includes: displaying the evaluation annotation content of the model to be evaluated in a preset manner.

[0088] The preset mode is a pre-set display mode, such as highlighting, bolding, coloring, etc. After the evaluation of the model to be evaluated is completed according to the evaluation strategy and its bound evaluation template, the generated evaluation annotation content is displayed in the preset mode to achieve the distinction between the evaluation annotation content and other content.

[0089] By displaying the evaluation annotation content in a preset manner, it is easier to locate the problem location more accurately and to summarize the evaluation results, providing a data basis for subsequent model training, thereby improving the iteration efficiency of the model.

[0090] In some optional implementations, the above method further includes:

[0091] Step S304, in response to the download operation for the evaluation annotation content, obtaining the evaluation annotation information of the model to be evaluated.

[0092] A download component is provided in the evaluation interface of the model evaluation system. After completing a round of evaluation tasks, the evaluator can click on the download component to download the evaluation annotation content corresponding to the evaluation task. Accordingly, the model evaluation system can respond to the evaluation technician's download operation on the evaluation annotation content, download the evaluation annotation content to the local computer, and parse the evaluation annotation content to extract the evaluation annotation information generated by the current round of evaluation tasks from the evaluation annotation content, wherein the evaluation annotation information may include the prediction results, actual execution results, error types, etc. of the model to be evaluated.

[0093] Step S305: Evaluate the performance of the model to be evaluated based on the evaluation annotation information to determine the evaluation result of the model to be evaluated.

[0094] Analyze the evaluation annotation information to determine the evaluation indicators of the model to be evaluated, which may include accuracy, recall, F1 value, etc. Evaluate the performance of the model to be evaluated in combination with the evaluation indicators to generate the evaluation results of the model to be evaluated. By analyzing the evaluation results, the model to be evaluated can be improved by adjusting model parameters, increasing the training data of the model to be evaluated, adjusting the model structure of the model to be evaluated, etc.

[0095] It should be noted that there can be multiple rounds of evaluation tasks, and the evaluation annotation content generated by each round of evaluation tasks can be downloaded locally, so as to comprehensively evaluate the model's performance based on the evaluation annotation information corresponding to multiple rounds of evaluation tasks.

[0096] By downloading the evaluation annotation content to extract the evaluation annotation information and evaluating the model performance based on the evaluation annotation information, it is convenient to provide a basis for subsequent model adjustments and to quickly complete the model training so that the model can achieve the expected training effect.

[0097] The model evaluation annotation method provided in this embodiment supports multiple evaluation methods in the evaluation strategy. Different evaluation methods correspond to different annotation templates. During the evaluation process, the required evaluation method and annotation template can be selected according to the actual evaluation requirements, thereby realizing model evaluation from multiple dimensions and further improving the evaluation accuracy of the model. For different annotation templates, corresponding evaluation annotations are performed to generate corresponding annotation results, thereby accurately locating the problem statement or problem link in combination with each annotation result, facilitating rapid troubleshooting of model problems for model adjustment, thereby facilitating improving model training efficiency and model iteration efficiency.

[0098] As an application example of the present disclosure, Figure 7 As shown in Figure 1, the evaluation technical team includes a first evaluation technician who writes the evaluation strategy, a second evaluation technician who determines the annotation template, and a third evaluation technician who annotates the evaluation task. The first evaluation technician can design an evaluation algorithm based on the evaluation direction of the model, and Figure 4 Write the evaluation strategy in the evaluation strategy editing interface shown. The second evaluation technician can unify the model annotation process according to the actual application scenario of the model, establish corresponding annotation specifications for different performances of the model, and create multiple template items to generate annotation templates. The first evaluation technician can bind the annotation template with the evaluation strategy according to actual business needs.

[0099] When the model needs to be evaluated, the evaluation strategy is triggered to initiate the evaluation task for the model, and the corresponding evaluation items are displayed in the evaluation interface according to the annotation template bound in the evaluation strategy. The third evaluation technician can mark the words of the text evaluation items, the modules of the link evaluation items, and the overall evaluation items for the model in the evaluation interface.

[0100] After completing the annotation for each evaluation item, the corresponding annotation results are generated and downloaded. Then, the evaluation annotation information is extracted from the annotation results to determine the model's misclassification information. Combined with the classification information, the model's evaluation conclusion is given.

[0101] The model evaluation annotation method provided in this embodiment can enable evaluation technicians to more accurately locate problematic sentences or problematic links in the model through word annotation of text evaluation items, module annotation of link evaluation items, and overall annotation of overall evaluation items, so as to facilitate evaluation technicians to summarize the evaluation results, provide data basis for subsequent training of the model, and greatly improve the iteration efficiency of the model.

[0102] In this embodiment, a model evaluation annotation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0103] This embodiment provides a model evaluation annotation device, such as Figure 8 As shown, including:

[0104] The acquisition module 401 is used to acquire the annotation template and the evaluation strategy of the model to be evaluated, and bind the annotation template to the evaluation strategy.

[0105] The evaluation display module 402 is used to display the evaluation interface of the evaluation task when the evaluation task corresponding to the evaluation strategy is triggered. The evaluation interface includes at least one evaluation item.

[0106] The annotation module 403 is used to generate an evaluation annotation content corresponding to the evaluation task in response to the annotation operation on at least one evaluation item.

[0107] In some optional implementations, the acquisition module 401 includes:

[0108] The evaluation requirement acquisition unit is used to obtain the evaluation requirement information of the model to be evaluated.

[0109] The template item creation unit is used to generate a marking template according to the template item creation operation in response to the template item creation operation based on the evaluation requirement information.

[0110] In some optional implementations, the acquisition module 401 further includes:

[0111] The strategy writing unit is used to respond to the strategy writing operation for the model to be evaluated and generate an evaluation strategy based on the strategy writing operation.

[0112] The template specifying unit is used to bind the annotation template to the evaluation strategy in response to a specifying operation on the annotation template in the evaluation strategy.

[0113] In some optional implementations, the acquisition module 401 further includes:

[0114] The annotation setting unit is used to determine the annotation range of the template item in response to the annotation setting operation on the template item.

[0115] In some optional implementations, the evaluation items include one or more of text evaluation items, link evaluation items, and overall evaluation items. Accordingly, the evaluation display module 402 includes:

[0116] The text evaluation unit is used to call the text annotation template corresponding to the text evaluation method if the evaluation strategy represents the text evaluation method of the model to be evaluated, and display the text evaluation items corresponding to the text annotation template on the evaluation interface.

[0117] The link evaluation unit is used to call the link annotation template corresponding to the execution link evaluation method if the evaluation strategy represents the execution link evaluation method of the model to be evaluated, and display the link evaluation items corresponding to the link annotation template on the evaluation interface.

[0118] The overall evaluation unit is used to call the overall annotation template corresponding to the overall evaluation method if the evaluation strategy represents the overall evaluation method of the model to be evaluated, and display the overall evaluation items corresponding to the overall annotation template on the evaluation interface.

[0119] In some optional implementations, the above-mentioned marking module 403 includes:

[0120] The first annotation unit is used to annotate the text information of the model to be evaluated in response to the annotation operation on the text evaluation item, and generate a first annotation result.

[0121] The second marking unit is used to respond to the marking operation on the link evaluation item, evaluate each module of the model to be evaluated, and generate a second marking result.

[0122] The third labeling unit is used to respond to the labeling operation on the overall evaluation item, perform overall evaluation on the model to be evaluated, and generate a third labeling result.

[0123] The annotation content generating unit is used to determine the evaluation annotation content of the model to be evaluated based on the first annotation result, and / or the second annotation result, and / or the third annotation result.

[0124] In some optional embodiments, the above device further includes:

[0125] The display module is used to display the evaluation annotation content of the model to be evaluated in a preset manner.

[0126] In some optional implementations, the above method further includes:

[0127] The download module is used to obtain the evaluation annotation information of the model to be evaluated in response to the download operation on the evaluation annotation content.

[0128] The evaluation module is used to evaluate the performance of the model to be evaluated based on the evaluation annotation information and determine the evaluation result of the model to be evaluated.

[0129] The annotation device for model evaluation in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0130] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0131] The model evaluation annotation device provided by the disclosed embodiment obtains the annotation template and evaluation strategy of the model to be evaluated, binds the evaluation strategy with the annotation template, and displays the corresponding evaluation interface in combination with the annotation template in the evaluation strategy when executing the evaluation task, so that the evaluation technician can evaluate and annotate the model according to the evaluation items of the annotation template displayed on the evaluation interface, thereby realizing standardized annotation for the model to be evaluated, avoiding the evaluation technicians from establishing their own standards and making it difficult to unify the evaluation results. Through the unified annotation specification, it is convenient to quickly determine the evaluation conclusion of the model to be evaluated according to the evaluation annotation content, improve the model evaluation annotation efficiency, and thus improve the model iteration efficiency.

[0132] The present disclosure also provides a computer device having the above Figure 8 Annotation setup for model evaluation shown.

[0133] See also Fig. 9 , Fig. 9 is a schematic diagram of a computer device provided by an optional embodiment of the present disclosure, such as Fig. 9As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig. 9 A processor 10 is taken as an example.

[0134] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0135] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0136] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0138] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0139] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0140] The computer device also includes a communication interface, which is used for the computer device to communicate with other devices or a communication network.

[0141] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0142] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0143] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A labeling method for model evaluation, characterized in that: The method comprises: Obtaining a labeling template and an evaluation strategy of the model to be evaluated, and binding the labeling template to the evaluation strategy; If the evaluation task corresponding to the evaluation strategy is triggered, the evaluation interface of the evaluation task is displayed, and the evaluation interface includes at least one evaluation item; In response to the annotation operation on the at least one evaluation item, evaluation annotation content corresponding to the evaluation task is generated.

2. The method according to claim 1, characterized in that Get the annotation template of the model to be evaluated, including: Obtaining evaluation requirement information of the model to be evaluated; In response to a template item creation operation based on the evaluation requirement information, the annotation template is generated according to the template item creation operation.

3. The method according to claim 2, characterized in that Also includes: In response to a tag setting operation for a template item, a tag range of the template item is determined.

4. The method according to claim 1 or 2, characterized in that: Get the evaluation strategy of the model to be evaluated, including: In response to a policy writing operation for the model to be evaluated, the evaluation policy is generated based on the policy writing operation.

5. The method according to claim 4, characterized in that The step of binding the annotation template to the evaluation strategy includes: In response to a specified operation on the annotation template in the evaluation strategy, the annotation template is bound to the evaluation strategy.

6. The method according to claim 1, characterized in that When the evaluation task corresponding to the evaluation strategy is triggered, the evaluation interface of the evaluation task is displayed, including: If the evaluation strategy represents the text evaluation method of the model to be evaluated, call the text annotation template corresponding to the text evaluation method, and display the text evaluation items corresponding to the text annotation template on the evaluation interface; and / or, If the evaluation strategy represents the execution link evaluation method of the model to be evaluated, call the link annotation template corresponding to the execution link evaluation method, and display the link evaluation items corresponding to the link annotation template on the evaluation interface; and / or, If the evaluation strategy represents the overall evaluation method of the model to be evaluated, call the overall annotation template corresponding to the overall evaluation method, and display the overall evaluation items corresponding to the overall annotation template on the evaluation interface; The evaluation item includes one or more of the text evaluation item, the link evaluation item, and the overall evaluation item.

7. The method according to claim 6, characterized in that The step of generating evaluation annotation content corresponding to the evaluation task in response to the annotation operation on the at least one evaluation item includes: In response to the annotation operation on the text evaluation item, annotating the text information of the model to be evaluated to generate a first annotation result; and / or, In response to the marking operation on the link evaluation item, each module of the model to be evaluated is evaluated to generate a second marking result; and / or, In response to the annotation operation on the overall evaluation item, the model to be evaluated is evaluated as a whole to generate a third annotation result; Based on the first annotation result, and / or the second annotation result, and / or the third annotation result, the evaluation annotation content of the model to be evaluated is determined.

8. The method according to claim 7, characterized in that Also includes: The evaluation annotation content of the model to be evaluated is displayed in a preset manner.

9. The method according to claim 1, characterized in that: Also includes: In response to a download operation on the evaluation annotation content, obtaining evaluation annotation information of the model to be evaluated; The performance capability of the model to be evaluated is evaluated according to the evaluation annotation information to determine the evaluation result of the model to be evaluated.

10. A labeling device for model evaluation, characterized in that: The device comprises: An acquisition module is used to acquire a labeling template and an evaluation strategy of a model to be evaluated, and bind the labeling template to the evaluation strategy; An evaluation display module, used for displaying an evaluation interface of the evaluation task when the evaluation task corresponding to the evaluation strategy is triggered, wherein the evaluation interface includes at least one evaluation item; The annotation module is used to generate the evaluation annotation content corresponding to the evaluation task in response to the annotation operation on the at least one evaluation item.

11. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the model evaluation labeling method according to any one of claims 1 to 9 by executing the computer instructions.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the model evaluation annotation method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to cause a computer to execute the annotation method for model evaluation according to any one of claims 1 to 9.

Images