Object evaluation method, device and equipment

Abstract

The embodiment of the invention provides an object evaluation method, device and equipment, and the method comprises the steps: receiving an annotation quality evaluation request for a to-be-evaluated target object; in response to the annotation quality evaluation request, acquiring an annotation task corresponding to a substandard annotation result in annotation results of the target object; according to the labeling task, generating test data, and obtaining a test result of the target object for the test data; determining a first evaluation result according to the task type of the labeling task and the labeling time of the labeling task processed by the target object; and according to the first evaluation result and the test result, determining an annotation quality evaluation result for the target object.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an object evaluation method, device and equipment. BACKGROUND

[0002] With the wide application of large models, the quality of data labeling has a particularly important influence on the performance and accuracy of the models, and the performance and accuracy of the models play a crucial role in protecting user privacy and ensuring data security.

[0003] To improve the quality of data labeling, labeling personnel can be regularly unified for examination, and labeling personnel who fail the examination can be retrained. However, due to the large number of labeling personnel and the uneven labeling quality of each labeling personnel, the examination accuracy of regular unified examination is low, and it is difficult to accurately evaluate the labeling quality of labeling personnel. Therefore, the technical solution for improving the labeling quality evaluation accuracy of labeling personnel is provided in the embodiments of the present application. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a technical solution for improving the labeling quality evaluation accuracy of labeling personnel.

[0005] To achieve the above technical solution, the embodiments of the present application are implemented as follows: The object evaluation method provided by the embodiments of the present application comprises the following steps: receiving a labeling quality evaluation request for a target object to be evaluated; in response to the labeling quality evaluation request, obtaining a labeling task corresponding to a labeling result that does not meet the standard in the labeling result of the target object; generating test data according to the labeling task, and obtaining a test result of the target object for the test data; determining a first evaluation result according to the task type of the labeling task and the labeling time of the target object for processing the labeling task; and determining a labeling quality evaluation result for the target object according to the first evaluation result and the test result.

[0006] The object evaluation device provided by the embodiments of the present application comprises the following modules: a request receiving module configured to receive a labeling quality evaluation request for a target object to be evaluated; a task obtaining module configured to, in response to the labeling quality evaluation request, obtain a labeling task corresponding to a labeling result that does not meet the standard in the labeling result of the target object; a test module configured to generate test data according to the labeling task, and obtain a test result of the target object for the test data; a first evaluation module configured to determine a first evaluation result according to the task type of the labeling task and the labeling time of the target object for processing the labeling task; and a second evaluation module configured to determine a labeling quality evaluation result for the target object according to the first evaluation result and the test result.

[0007] An object evaluation device provided by an embodiment of the present specification includes a processor and a memory arranged to store computer executable instructions that, when executed, cause the processor to: receive a labeling quality evaluation request for a target object to be evaluated; in response to the labeling quality evaluation request, obtain a labeling task corresponding to a labeling result that does not meet a standard in labeling results of the target object; generate test data according to the labeling task, and obtain a test result of the target object for the test data; determine a first evaluation result according to a task type of the labeling task and a labeling time of the target object processing the labeling task; and determine a labeling quality evaluation result for the target object according to the first evaluation result and the test result.

[0008] An embodiment of the present specification also provides a storage medium for storing computer executable instructions, which, when executed by a processor, implement the following processes: receiving a labeling quality evaluation request for a target object to be evaluated; in response to the labeling quality evaluation request, obtaining a labeling task corresponding to a labeling result that does not meet a standard in labeling results of the target object; generating test data according to the labeling task, and obtaining a test result of the target object for the test data; determining a first evaluation result according to a task type of the labeling task and a labeling time of the target object processing the labeling task; and determining a labeling quality evaluation result for the target object according to the first evaluation result and the test result.

[0009] An embodiment of the present specification also provides a computer program product including a computer program that, when executed by a processor, implements the following processes: receiving a labeling quality evaluation request for a target object to be evaluated; in response to the labeling quality evaluation request, obtaining a labeling task corresponding to a labeling result that does not meet a standard in labeling results of the target object; generating test data according to the labeling task, and obtaining a test result of the target object for the test data; determining a first evaluation result according to a task type of the labeling task and a labeling time of the target object processing the labeling task; and determining a labeling quality evaluation result for the target object according to the first evaluation result and the test result. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only represent some embodiments described in the present specification, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings. Figure 1 A schematic diagram of an object evaluation method of the present specification; Figure 2 A schematic diagram of a target object determination process of the present specification; Figure 3 A schematic diagram of a test data generation process of the present specification; Figure 4 A schematic diagram of a first evaluation result determination process of the present specification; Figure 5 A schematic diagram of an object evaluation process of the present specification; Figure 6 A schematic diagram of a target labeling task processing process of the present specification; Figure 7 A schematic diagram of an object evaluation device of the present specification; Figure 8 A schematic diagram of an object evaluation apparatus of the present specification. DETAILED DESCRIPTION

[0011] The embodiments of the present specification provide an object evaluation method, device and apparatus.

[0012] In order for those skilled in the art to better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be described clearly and completely in conjunction with the drawings in the embodiments of the present specification. Obviously, the described embodiments only represent some embodiments of the present specification, rather than all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present specification.

[0013] The embodiment of the present specification provides a technical solution for improving the accuracy of label quality evaluation of labelers. The quality of data labeling has a significant impact on the performance and accuracy of the model, and the performance and accuracy of the model play a crucial role in protecting user privacy and ensuring data security. To improve the quality of data labeling, labelers can be regularly evaluated, and labelers who fail the evaluation can be retrained. However, due to the large number of labelers and the uneven quality of each labeler's labeling, regular unified evaluation has low accuracy and cannot accurately evaluate the labeling quality of labelers. In this solution, a labeling quality evaluation request for a target object to be evaluated is received, and in response to the labeling quality evaluation request, the labeling tasks corresponding to the labeling results that do not meet the standards in the labeling results of the target object are obtained. According to the labeling task, test data is generated, and the test results of the target object for the test data are obtained. According to the task type of the labeling task and the labeling time of the target object processing the labeling task, a first evaluation result is determined, and according to the first evaluation result and the test result, a labeling quality evaluation result for the target object is determined. In this way, first, since the labeling problems of labelers may not be the same, the test data generated by the labeling tasks corresponding to the labeling results that do not meet the standards in the labeling results of the target object can be targeted to test the target object, avoiding the problem of low accuracy of labeling quality evaluation in unified evaluation. Secondly, based on the test result, in combination with the first evaluation result determined by the task type of the labeling task and the labeling time of the target object processing the labeling task, the target object can be comprehensively evaluated to obtain an accurate labeling quality evaluation result. Specific processing can be referred to the specific content in the following embodiments.

[0014] As shown in Figure 1 The embodiment of the present specification provides an object evaluation method, and the execution subject of the method can be a server. The server can be an independent server or a server cluster composed of multiple servers. The server can be a background server of a financial service or an online shopping service, or a background server of an application program. In this embodiment, the execution subject is taken as an example to be a server, and the method can specifically include the following steps: In step S102, a labeling quality evaluation request for a target object to be evaluated is received.

[0015] The target object can be any object capable of processing labeling tasks, such as a labeler or a labeling model. The labeling task can be a task of adding structured labels (such as classification labels, entity labels, and sentiment orientation) to text, image, audio, and video data. The labeling result can be used to enable the model to learn the mapping relationship between the input data and the output data.

[0016] In implementation, taking the target object as an example, the server can trigger the labeling quality evaluation request for the target object to be evaluated based on a preset evaluation period (such as nearly half a month, nearly one month, etc.).

[0017] Alternatively, the server can also obtain the completion condition of the labeling task with a priority higher than the preset priority threshold in the preset evaluation period, and determine whether to trigger the labeling quality evaluation request for the labeling personnel corresponding to the labeling task according to the completion condition. For example, in the preset evaluation period, the priority of the labeling task 1 is higher than the preset priority threshold, and the training effect of the subsequent model training based on the labeling result of the labeling task 1 does not meet the training requirement, then it can be considered that the completion condition of the labeling task 1 is not up to standard, and the server can determine the labeling personnel processing the labeling task 1 as the target object, and trigger the labeling quality evaluation request for the target object.

[0018] In addition, the triggering mode of the above labeling quality evaluation request is an optional and implementable triggering mode. In actual application scenarios, there can be many different triggering modes, and different triggering modes can be selected according to different actual application scenarios. The embodiments of the present specification do not make specific limitations on this.

[0019] In step S104, in response to the labeling quality evaluation request, the labeling task corresponding to the labeling result not up to standard in the labeling result of the target object is obtained.

[0020] The labeling task can be any task that needs to be labeled, for example, the labeling task can be to determine the risk type of the resource transfer data in a preset detection period, that is, to determine the risk label (such as high risk, medium risk, low risk, etc.) corresponding to the resource transfer data, or the labeling task can also be to determine the user type of the platform registered user, that is, to determine the type label (such as high-risk-aware user, low-risk-aware user, etc.) of the platform registered user.

[0021] In implementation, when the labeling task is not up to standard, the server can determine the labeling result corresponding to the labeling task as not up to standard. For example, assuming that the labeling task is to determine the risk label of 100 pieces of resource transfer data in the past week, the server can obtain the labeling result corresponding to the labeling task according to the labeling processing of the resource transfer data in the labeling task by multiple labeling personnel, and train the preset risk detection model according to the labeling task and the labeling result. If the test effect of the trained risk detection model does not meet the preset detection requirement, it can be determined that the labeling task is not up to standard, and the labeling result of the 100 pieces of resource transfer data can be determined as the labeling result not up to standard.

[0022] Alternatively, the server can also send the same annotation task to multiple different annotators for annotation, and can determine the compliance of the annotation result of each annotator according to the annotation results of the multiple different annotators for the same annotation task. For example, the server can perform clustering processing on the annotation results of the multiple different annotators for the same annotation task, determine the final annotation result according to the number of annotation results contained in each class, and determine the compliance of the annotation result of each annotator according to the matching relationship between the final annotation result and the annotation result of each annotator.

[0023] In addition, the above-mentioned method for determining the compliance of the annotation result is an optional and implementable determination method. In actual application scenarios, there can be various different determination methods, and different determination methods can be selected according to different actual application scenarios. The embodiments of the present specification do not make specific limitations in this regard.

[0024] In step S106, test data is generated according to the annotation task, and a test result of the target object for the test data is obtained.

[0025] In implementation, the server can determine test information corresponding to the task type of the annotation task according to a preset correspondence between the task type and the test information, and generate test data according to the test information.

[0026] Alternatively, the server can also generate test data according to the annotation task by using a pre-trained test data generation model.

[0027] In addition, the above-mentioned method for generating test data is an optional and implementable determination method. In actual application scenarios, there can be various different generation methods, and different generation methods can be selected according to different actual application scenarios. The embodiments of the present specification do not make specific limitations in this regard.

[0028] The server can send the test data to the target object, and obtain a test result of the target object for the test data. For example, the test data can include a predetermined number of test questions, and the server can determine the test result of the target object for the test data according to the answers submitted by the target object for the test questions and the predetermined answers. Alternatively, the test data can also include a predetermined number of data to be labeled, and the server can determine the test result of the target object for the test data according to the compliance of the target object for each data to be labeled and the preset label data.

[0029] The test result can include test pass, test fail, etc., or the test result can also include test scores, etc.

[0030] In step S108, the first evaluation result is determined according to the task type of the labeling task and the labeling time of the target object processing the labeling task.

[0031] In implementation, the server can determine the time threshold corresponding to the labeling task of different task types according to the preset correspondence between the task type and the time threshold, and determine the first evaluation result according to the time threshold and the labeling time of the target object processing the corresponding labeling task.

[0032] The first evaluation result can include non-standard and standard, etc., such as the server can determine the first evaluation result as standard or non-standard according to the average value of the difference between the time threshold and the labeling time of the target object processing the corresponding labeling task.

[0033] Alternatively, the first evaluation result can also be an evaluation score, and the server can determine the evaluation score according to the absolute value of the difference between the time threshold and the labeling time of the target object processing the corresponding labeling task.

[0034] Alternatively, the server can obtain the labeling time of the labeling task corresponding to the same task type, and determine the first evaluation result according to the obtained labeling time and the labeling time of the target object processing the labeling task corresponding to the task type.

[0035] In addition, the above-mentioned determination method of the first evaluation result is an optional and implementable determination method, and in actual application scenarios, there can be many different determination methods, which can be selected according to the difference of actual application scenarios, and the embodiments of the present specification do not make specific limitation.

[0036] In step S110, the labeling quality evaluation result for the target object is determined according to the first evaluation result and the test result.

[0037] The labeling quality evaluation result can include qualified and unqualified, or the labeling quality evaluation result can also include a labeling quality score, a labeling quality type (such as high quality, low quality, etc.), etc.

[0038] In implementation, in the case that the first evaluation result is standard (or the evaluation score is greater than a preset evaluation threshold, etc.), and the test result is test passed (or the test score is greater than a preset test threshold), the labeling quality evaluation result for the target object can be determined as qualified (or a labeling quality type of high quality).

[0039] In the case that the first evaluation result is non-standard (or the evaluation score is greater than a preset evaluation threshold, etc.), or the test result is test failed (or the test score is not greater than a preset test threshold), the labeling quality evaluation result for the target object can be determined as unqualified (or a labeling quality type of low quality).

[0040] In addition, the determination method of the labeling quality evaluation result can be various, and different determination methods can be selected according to different actual application scenarios, which is not limited in the embodiments of the present specification.

[0041] The embodiments of the present specification provide an object evaluation method, which receives a labeling quality evaluation request for a target object to be evaluated, acquires a labeling task corresponding to a labeling result that does not meet a standard in a labeling result of the target object in response to the labeling quality evaluation request, generates test data according to the labeling task, acquires a test result of the target object for the test data, determines a first evaluation result according to a task type of the labeling task and a labeling time of the target object for processing the labeling task, and determines a labeling quality evaluation result for the target object according to the first evaluation result and the test result. In this way, first, since the labeling problems of the labeling personnel can not be the same, the test data generated by the labeling task corresponding to the labeling result that does not meet the standard in the labeling result of the target object can be used to specifically label the target object, thereby avoiding the problem of low labeling quality evaluation accuracy in unified examination. Secondly, on the basis of the test result, the first evaluation result determined by the task type of the labeling task and the labeling time of the target object for processing the labeling task can be used to comprehensively examine the target object to obtain an accurate labeling quality evaluation result.

[0042] In actual application, the target object can also be selected from the candidate objects, and the determination method of the target object can be various. The following provides an optional determination method, as shown in the following. Figure 2 The specific process can include the following steps S202-S206.

[0043] In step S202, the labeling result of the candidate object in a preset detection period is acquired.

[0044] The preset detection period can be nearly half a month, nearly three months, etc.

[0045] In step S204, the correct rate corresponding to the candidate object is determined according to the proportion of the labeling result that does not meet the standard in the labeling result.

[0046] In implementation, the server can determine the proportion of the labeling result that does not meet the standard in the labeling result as the correct rate corresponding to the candidate object.

[0047] In step S206, the candidate object is selected according to the correct rate, and the selected candidate object is determined as the target object.

[0048] In implementation, the server can determine the candidate object with a correct rate less than a preset correct rate threshold as the target object.

[0049] In actual applications, the specific processing manner of generating the test data according to the labeling task in the step S106 can be various, and an optional processing manner is provided as follows, for example. Figure 3 As shown in the step S1062, the specific processing can include the following steps A1-A3.

[0050] In the step S1062, the test data is generated according to the task type of the labeling task, the error type to which the labeling result of the target object for the labeling task belongs, and the type of the labeling group to which the target object belongs.

[0051] The task type of the labeling task can include a picture labeling type, a video labeling type, a text labeling type, a risk labeling type, a user type labeling type, a structured data labeling type, an unstructured data labeling type, a quality inspection task, an acceptance task, etc., the error type can include insufficient comprehensive abnormality analysis, missing label selection, incomplete key abnormality detection, and non-specific recommended information, and the type of the labeling group to which the target object belongs can include a quality inspection group and a training data labeling group.

[0052] In actual applications, the specific processing manner of generating the test data according to the task type of the labeling task, the error type to which the labeling result of the target object for the labeling task belongs, and the type of the labeling group to which the target object belongs in the step S1062 can be various, and an optional processing manner is provided as follows, for example. Figure 3 As shown in the step S1062, the specific processing can include the following steps A1-A3.

[0053] In the step A1, the error type to which the labeling result of the target object for the labeling task belongs is determined.

[0054] In implementation, the preset large language model is used to determine the error type to which the labeling result of the target object for the labeling task belongs according to the labeling result of the target object for the labeling task and the target labeling result corresponding to the labeling task.

[0055] The target labeling result is a result determined by a first object for labeling the labeling task, and the labeling quality evaluation result of the first object is higher than a preset quality threshold.

[0056] In implementation, the server can determine the first object in the target object according to the labeling quality evaluation result of each target object obtained in the last round of object evaluation processing, and determine the target labeling result according to the labeling result of the first object for the labeling task, so as to determine the error type to which the labeling result of the target object for the labeling task belongs in the current object evaluation processing according to the target labeling result.

[0057] In the step A2, the proportion corresponding to each error type is determined according to the number of labeling tasks corresponding to each error type and the number of labeling tasks.

[0058] In step A3, test data is generated based on the task type of the annotation task, the proportion of each error type, and the type of the annotation group to which the target object belongs.

[0059] In practice, by using the proportion of each error type, targeted test data that matches the annotation capabilities of the target object can be generated, thereby improving the accuracy of the evaluation of the annotation quality of the target object.

[0060] In practical applications, the specific processing method for determining the first evaluation result in step S108 above, based on the task type of the annotation task and the annotation time of the target object, can vary. One optional processing method is provided below, such as... Figure 4 As shown, the specific process may include the following steps, S1082.

[0061] In step S1082, a first evaluation result is determined using a pre-trained quality assessment model based on the task type of the annotation task, the error type of the annotation result of the target object for the annotation task, and the annotation time of the target object in processing the annotation task.

[0062] The quality assessment model can be a model built based on a preset deep learning algorithm.

[0063] In implementation, for example, the server can build a quality assessment model based on a neural network algorithm and train the assessment model with historical data. The trained quality assessment model can then determine the first assessment result based on the task type of the annotation task, the error type of the annotation result of the target object for the annotation task, and the annotation time of the target object in processing the annotation task.

[0064] The annotation task can be distributed to different annotators for initial annotation. After the annotators complete the annotation, it can enter the inspection stage. Senior quality inspectors (i.e., the first target) can randomly select samples to check the accuracy of this annotation task. After the internal inspection is completed, it is released to the acceptance party for acceptance of the overall data.

[0065] like Figure 5 As shown, errors identified during inspection and acceptance (i.e., substandard annotation results) can be fed back online to the Intelligent Annotation (ITAG) Error Center. Errors generated during quality inspection and / or acceptance can be automatically analyzed based on the question, personnel, personnel group, and error integration, generating comprehensive reports to pinpoint problem areas and facilitate rapid improvement.

[0066] The management party can generate a data set by customizing quick selection questions, circling and labeling personnel (i.e., determining target objects), and communicating with the examination center to generate examination questions (i.e., test data), without the need to download and upload offline again.

[0067] After the target object passes the examination center's wrong learning examination (i.e., obtains the test result), if it passes the examination (i.e., the test result meets the requirements), it is determined that the target object passes the test and can continue to process the labeling task. If it does not pass the examination (i.e., the test result does not meet the requirements), the target object needs to be trained, etc.

[0068] The above wrong question attribution and examination are short-term solutions. Useful features can be extracted from wrong questions, such as question type, error type, and answer time, and the features can be effectively selected and engineered. Finally, the performance of the labeling personnel can be determined based on the first evaluation result and the test result, thereby providing a reference for personnel selection and elimination.

[0069] In this way, the above object evaluation process can solve the problems of long evaluation process, low link convenience, and limited wrong question application solutions. By establishing an online wrong question center, reflowing inspection and acceptance of wrong question cases, automatically generating analysis big plates, and connecting with the examination center, the problem points can be quickly located and the quality can be improved. At the same time, the wrong question set can be abstracted into features, which can improve the accuracy of predicting the quality of the labeling personnel.

[0070] In actual application, the target object for the target labeling task can also be determined according to the labeling quality evaluation result of the target object. The specific processing method of the target labeling task can be various, and one optional processing method is provided as follows. Figure 6 As shown in the figure, the specific processing can include the following steps S602.

[0071] In step S602, in the case of receiving a labeling request for a target labeling task, the target object is selected and processed according to the labeling quality evaluation result of the target object, and the target labeling task is sent to the selected target object for labeling processing.

[0072] In implementation, the server can select and process the target object according to the priority of the target labeling task, the task type of the target labeling task, the labeling quality evaluation result of the target object, and the number of tasks that the target object has not completed, and send the target labeling task to the selected target object for labeling processing.

[0073] The embodiment of the present specification provides an object evaluation method, by receiving a labeling quality evaluation request for a target object to be evaluated, in response to the labeling quality evaluation request, obtaining the labeling task corresponding to the labeling result that does not meet the standard in the labeling result of the target object, generating test data according to the labeling task, and obtaining the test result of the target object for the test data, determining the first evaluation result according to the task type of the labeling task and the labeling time of the target object processing the labeling task, and determining the labeling quality evaluation result for the target object according to the first evaluation result and the test result. In this way, first, since the labeling problems of the labeling personnel may not be the same, the test data generated by the labeling task corresponding to the labeling result that does not meet the standard in the labeling result of the target object can be targeted for labeling test of the target object, avoiding the problem of low labeling quality evaluation accuracy in unified examination, and second, on the basis of the test result, in combination with the first evaluation result determined by the task type of the labeling task and the labeling time of the target object processing the labeling task, the target object can be comprehensively examined to obtain an accurate labeling quality evaluation result.

[0074] The object evaluation method provided by the embodiment of the present specification is based on the same idea, and the embodiment of the present specification also provides an object evaluation device, as shown in Figure 7 .

[0075] The object evaluation device comprises a request receiving module 701, a task obtaining module 702, a test module 703, a first evaluation module 704 and a second evaluation module 705, wherein: The request receiving module 701 is configured to receive a labeling quality evaluation request for a target object to be evaluated; The task obtaining module 702 is configured to obtain, in response to the labeling quality evaluation request, the labeling task corresponding to the labeling result that does not meet the standard in the labeling result of the target object; The test module 703 is configured to generate test data according to the labeling task, and obtain the test result of the target object for the test data; The first evaluation module 704 is configured to determine a first evaluation result according to the task type of the labeling task and the labeling time of the target object processing the labeling task; The second evaluation module 705 is configured to determine a labeling quality evaluation result for the target object according to the first evaluation result and the test result.

[0076] In the embodiment of the present specification, the device further comprises: The result obtaining module is configured to obtain the labeling result of the candidate object in a preset detection period; The correctness determination module is configured to determine a correctness of the candidate object according to a proportion of the labeling results that do not meet the standard in the labeling results. The object screening module is configured to perform screening processing on the candidate object according to the correctness, and determine the target object as the candidate object after the screening processing.

[0077] In the embodiments of the present specification, the test module 703 is configured to: generate the test data according to a task type of the labeling task, an error type to which the labeling result of the target object for the labeling task belongs, and a type of a labeling group to which the target object belongs.

[0078] In the embodiments of the present specification, the test module 703 is configured to: determine an error type to which the labeling result of the target object for the labeling task belongs; determine a proportion corresponding to each error type according to a number of the labeling task corresponding to each error type and a number of the labeling task; generate the test data according to a task type of the labeling task, the proportion corresponding to each error type, and a type of a labeling group to which the target object belongs.

[0079] In the embodiments of the present specification, the test module 703 is configured to: determine, by using a preset large language model, an error type to which the labeling result of the target object for the labeling task belongs, according to the labeling result of the target object for the labeling task and a target labeling result corresponding to the labeling task.

[0080] In the embodiments of the present specification, the target labeling result corresponding to the labeling task is a result determined by a first object for labeling the labeling task, and a labeling quality evaluation result of the first object is higher than a preset quality threshold.

[0081] In the embodiments of the present specification, the first evaluation module 704 is configured to: determine the first evaluation result according to a task type of the labeling task, an error type to which the labeling result of the target object for the labeling task belongs, and a labeling time of the target object for processing the labeling task, by using a pre-trained quality evaluation model, wherein the quality evaluation model is a model constructed based on a preset deep learning algorithm. In the embodiments of the present specification, the apparatus further includes: The object screening module is configured to, in a case where a labeling request for a target labeling task is received, perform screening processing on the target object according to the labeling quality evaluation result of the target object, and send the target labeling task to the target object after the screening processing for labeling processing.

[0082] The object evaluation device provided by the embodiments of the present specification can receive a labeling quality evaluation request for a target object to be evaluated, obtain a labeling task corresponding to a labeling result that does not meet a standard in a labeling result of the target object in response to the labeling quality evaluation request, generate test data according to the labeling task, obtain a test result of the target object for the test data, determine a first evaluation result according to a task type of the labeling task and a labeling time of the target object for processing the labeling task, and determine a labeling quality evaluation result for the target object according to the first evaluation result and the test result. In this way, first, since the labeling problems of the labeling personnel can not be the same, the test data generated by the labeling task corresponding to the labeling result that does not meet the standard in the labeling result of the target object can be used to specifically test the labeling of the target object, thereby avoiding the problem of low accuracy of labeling quality evaluation in unified examination. Second, on the basis of the test result, the first evaluation result determined by the task type of the labeling task and the labeling time of the target object for processing the labeling task can be used to comprehensively evaluate the target object to obtain an accurate labeling quality evaluation result.

[0083] The object evaluation device provided by the embodiments of the present specification is based on the same idea. The embodiments of the present specification also provide an object evaluation device, as shown in Figure 8 .

[0084] The object evaluation device can be a terminal device or a server provided by the above embodiments.

[0085] The object evaluation device can have great differences due to different configurations or performances, and can include one or more processors 801 and memories 802. The memory 802 can store one or more storage applications or data. The memory 802 can be temporary storage or persistent storage. The applications stored in the memory 802 can include one or more modules (not shown in the figure), and each module can include a series of computer executable instructions in the object evaluation device. Furthermore, the processor 801 can be configured to communicate with the memory 802 and execute a series of computer executable instructions in the memory 802 on the object evaluation device. The object evaluation device can also include one or more power supplies 803, one or more wired or wireless network interfaces 804, one or more input and output interfaces 805, and one or more keyboards 806.

[0086] In particular embodiments, the object evaluation device includes a memory, and one or more programs, wherein one or more programs are stored in the memory and the one or more programs can include one or more modules, and each module can include a series of computer-executable instructions in the object evaluation device, and the one or more programs configured to be executed by one or more processors include computer-executable instructions for performing the following: receiving a labeling quality evaluation request for a target object to be evaluated; in response to the labeling quality evaluation request, obtaining a labeling task corresponding to a labeling result that does not meet a standard in labeling results of the target object; generating test data according to the labeling task, and obtaining a test result of the target object for the test data; determining a first evaluation result according to a task type of the labeling task and a labeling time of the target object processing the labeling task; determining a labeling quality evaluation result for the target object according to the first evaluation result and the test result.

[0087] Each of the embodiments in the specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the object evaluation device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0088] The object evaluation device provided by the embodiment of the specification receives a labeling quality evaluation request for a target object to be evaluated, in response to the labeling quality evaluation request, obtains a labeling task corresponding to a labeling result that does not meet a standard in labeling results of the target object, generates test data according to the labeling task, and obtains a test result of the target object for the test data, determines a first evaluation result according to a task type of the labeling task and a labeling time of the target object processing the labeling task, and determines a labeling quality evaluation result for the target object according to the first evaluation result and the test result. In this way, first, since the labeling problems of the labeling personnel can not be the same, the test data generated by the labeling task corresponding to the labeling result that does not meet the standard in the labeling results of the target object can be targeted for labeling test of the target object, avoiding the problem of low accuracy of labeling quality evaluation in unified examination. Secondly, on the basis of the test result, in combination with the first evaluation result determined by the task type of the labeling task and the labeling time of the target object processing the labeling task, the target object can be comprehensively evaluated to obtain an accurate labeling quality evaluation result.

[0089] Further, based on the above Figures 1 to 6 One or more embodiments of the present specification also provide a storage medium for storing computer executable instruction information, in a specific embodiment, the storage medium can be a U disk, an optical disk, a hard disk, etc. The computer executable instruction information stored in the storage medium can implement the following processes when executed by a processor: receiving a labeling quality evaluation request for a target object to be evaluated; In response to the labeling quality evaluation request, obtaining a labeling task corresponding to a labeling result that does not meet the standard in the labeling result of the target object; According to the labeling task, generate test data, and obtain the test result of the target object for the test data; According to the task type of the labeling task and the labeling time of the target object processing the labeling task, determine the first evaluation result; According to the first evaluation result and the test result, determine the labeling quality evaluation result for the target object.

[0090] Each embodiment in the present specification is described in a progressive manner, and the same and similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the above-mentioned storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts can be referred to the part of the method embodiment.

[0091] The embodiment of the present specification provides a storage medium, by receiving a labeling quality evaluation request for a target object to be evaluated, in response to the labeling quality evaluation request, obtaining a labeling task corresponding to a labeling result that does not meet the standard in the labeling result of the target object, according to the labeling task, generating test data, and obtaining the test result of the target object for the test data, according to the task type of the labeling task and the labeling time of the target object processing the labeling task, determine the first evaluation result, according to the first evaluation result and the test result, determine the labeling quality evaluation result for the target object. In this way, first, since the labeling problems of the labeling personnel can not be the same, therefore, by the labeling task corresponding to the labeling result that does not meet the standard in the labeling result of the target object, the test data generated can be targeted to the labeling test of the target object, avoiding the problem of low accuracy of labeling quality evaluation in unified examination, second, on the basis of the test result, in combination with the first evaluation result determined by the task type of the labeling task and the labeling time of the target object processing the labeling task, the target object can be comprehensively evaluated to obtain an accurate labeling quality evaluation result.

[0092] Further, based on the above Figures 1 to 6The one or more embodiments of the specification also provide a computer program product comprising a computer program, the computer program in the computer program product being capable of implementing the following flow when executed by a processor: receiving a labeling quality evaluation request for a target object to be evaluated; In response to the labeling quality evaluation request, obtaining a labeling task corresponding to a labeling result that does not meet a standard in the labeling result of the target object; According to the labeling task, generating test data, and obtaining a test result of the target object for the test data; According to the task type of the labeling task and the labeling time of the target object processing the labeling task, determining a first evaluation result; According to the first evaluation result and the test result, determining a labeling quality evaluation result for the target object.

[0093] Each of the embodiments in the specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the above-mentioned computer program product embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0094] The embodiment of the specification provides a computer program product, which receives a labeling quality evaluation request for a target object to be evaluated, in response to the labeling quality evaluation request, obtains a labeling task corresponding to a labeling result that does not meet a standard in the labeling result of the target object, according to the labeling task, generates test data, and obtains a test result of the target object for the test data, according to the task type of the labeling task and the labeling time of the target object processing the labeling task, determines a first evaluation result, and according to the first evaluation result and the test result, determines a labeling quality evaluation result for the target object. In this way, first, since the labeling problems of the labeling personnel may not be the same, the test data generated by the labeling task corresponding to the labeling result that does not meet the standard in the labeling result of the target object can be targeted to test the target object, avoiding the problem of low accuracy of labeling quality evaluation in unified examination. Secondly, on the basis of the test result, in combination with the first evaluation result determined by the task type of the labeling task and the labeling time of the target object processing the labeling task, the target object can be comprehensively evaluated to obtain an accurate labeling quality evaluation result.

[0095] The above described embodiments of the present description have been described. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0096] In the 1990s, it was possible to distinguish whether an improvement in a technology was a hardware improvement (e.g., an improvement in the circuit structure of a diode, transistor, switch, etc.) or a software improvement (an improvement in a method flow). However, as technology has advanced, many improvements in method flows today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into a hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming it themselves, without having to ask a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating an integrated circuit chip, this programming is now mostly implemented using "logic compiler" software, which is similar to the software compiler used when developing a program, and the original code before compilation must also be written in a specific programming language, which is called a hardware description language (HDL), and there are many types of HDL, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that it is very easy to obtain a hardware circuit that implements a logical method flow by simply logically programming the method flow in one of the above-mentioned hardware description languages and programming it into an integrated circuit.

[0097] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the microprocessor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to implementing the controller in pure computer readable program code, it is possible to implement the same functionality in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Such a controller can therefore be considered to be a hardware component, and the means included therein for implementing the various functions can also be considered to be structures within the hardware component. Alternatively, or even additionally, the means for implementing the various functions can be considered to be both a software module implementing the method and a structure within the hardware component.

[0098] The systems, apparatuses, modules or units illustrated by the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0099] For the sake of description, the above apparatuses are described in functional division and are described respectively. Of course, the functions of each unit can be implemented in the same or multiple software and / or hardware when implementing one or more embodiments of the present specification.

[0100] Those skilled in the art will understand that the embodiments of the present specification can be provided as a method, a system, or a computer program product. Therefore, one or more embodiments of the present specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] The embodiments of the present specification are described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present specification. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable electronic devices to produce a machine, so that the instructions executed by the computer or other programmable electronic devices generate a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks. Figure 1 one or more flows and / or blocks.

[0102] These computer program instructions can also be stored in a computer readable memory capable of directing the computer or other programmable electronic devices to work in a specific way, so that the instructions stored in the computer readable memory produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks. Figure 1 one or more flows and / or blocks.

[0103] These computer program instructions can also be loaded into a computer or other programmable electronic devices, so that a series of operation steps are performed on the computer or other programmable electronic devices to produce a computer implemented process, so that the instructions executed on the computer or other programmable electronic devices provide steps for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks. Figure 1 one or more flows and / or blocks.

[0104] In a typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces and memories.

[0105] The memory can include non-persistent memory in the computer readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of the computer readable medium.

[0106] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0107] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0108] Those skilled in the art will appreciate that embodiments of the present specification can be provided as methods, systems or computer program products. Therefore, one or more embodiments of the present specification can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] One or more embodiments of the present specification can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. One or more embodiments of the present specification can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media, including storage devices.

[0110] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0111] The above only describes the embodiments of the specification and is not used to limit the file. The specification can have various changes and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the specification shall be included in the scope of claims of the specification.

Claims

1. A method for object evaluation, comprising: receiving a request for evaluation of a target object; in response to the request for evaluation of the target object, obtaining a labeling task corresponding to a labeling result that does not meet a standard in a labeling result of the target object; generating test data according to the labeling task, and obtaining a test result of the target object for the test data; determining a first evaluation result according to a task type of the labeling task and a labeling time of the target object for processing the labeling task; determining a labeling quality evaluation result of the target object according to the first evaluation result and the test result.

2. The method of claim 1, before the receiving a request for evaluation of a target object, further comprising: obtaining a labeling result of a candidate object in a preset detection period; determining a correctness rate of the candidate object according to a proportion of labeling results that do not meet a standard in the labeling result; performing a screening process on the candidate object according to the correctness rate, and determining a screened candidate object as the target object.

3. The method of claim 1, wherein the generating test data according to the labeling task comprises: generating the test data according to a task type of the labeling task, an error type of a labeling result of the target object for the labeling task, and a type of a labeling group to which the target object belongs.

4. The method of claim 3, wherein the generating test data according to a task type of the labeling task, an error type of a labeling result of the target object for the labeling task, and a type of a labeling group to which the target object belongs comprises: determining an error type of a labeling result of the target object for the labeling task; determining a proportion corresponding to each error type according to a number of labeling tasks corresponding to each error type and a number of labeling tasks; generating the test data according to a task type of the labeling task, a proportion corresponding to each error type, and a type of a labeling group to which the target object belongs.

5. The method of claim 4, wherein the determining an error type of a labeling result of the target object for the labeling task comprises: determining an error type of a labeling result of the target object for the labeling task according to a labeling result of the target object for the labeling task and a target labeling result corresponding to the labeling task by using a preset large language model.

6. The method of claim 5, wherein the target labeling result corresponding to the labeling task is a result determined by a first object for labeling the labeling task, and a labeling quality evaluation result of the first object is higher than a preset quality threshold.

7. The method of claim 6, wherein the determining a first evaluation result according to a task type of the labeling task and a labeling time of the target object for processing the labeling task comprises: The first evaluation result is determined by using a pre-trained quality evaluation model, according to a task type of the labeling task, an error type to which a labeling result of the target object for the labeling task belongs, and a labeling time of the target object for processing the labeling task.

8. The method of claim 1, further comprising: In a case where a labeling request for a target labeling task is received, performing screening processing on the target object according to the labeling quality evaluation result of the target object, and sending the target labeling task to the screened target object for labeling processing.

9. An object evaluation apparatus comprising: a request receiving module configured to receive a labeling quality evaluation request for a target object to be evaluated; a task obtaining module configured to, in response to the labeling quality evaluation request, obtain a labeling task corresponding to a labeling result that does not meet a standard among labeling results of the target object; a testing module configured to generate test data according to the labeling task, and obtain a test result of the target object for the test data; a first evaluation module configured to determine a first evaluation result according to a task type of the labeling task and a labeling time of the target object for processing the labeling task; a second evaluation module configured to determine a labeling quality evaluation result for the target object according to the first evaluation result and the test result.

10. An object evaluation apparatus comprising: a processor; and a memory arranged to store computer-executable instructions that, when executed, cause the processor to: receive a labeling quality evaluation request for a target object to be evaluated; in response to the labeling quality evaluation request, obtain a labeling task corresponding to a labeling result that does not meet a standard among labeling results of the target object; generate test data according to the labeling task, and obtain a test result of the target object for the test data; determine a first evaluation result according to a task type of the labeling task and a labeling time of the target object for processing the labeling task; determine a labeling quality evaluation result for the target object according to the first evaluation result and the test result.

Images