Labeling task allocation method, related device, equipment and storage medium

Abstract

The invention discloses an annotation task allocation method, a related device, equipment and a storage medium, and the method comprises the steps: obtaining the index data of a target task type executed by a candidate annotation customer based on the target task type of a to-be-allocated annotation task; wherein the index data comprises index values of a plurality of index dimensions; obtaining a weight factor of each index dimension under the target task type based on the configuration information about the relative emphasis of the plurality of index dimensions under the target task type; weighting based on the weight factor and the index value of the same index dimension under the target task type to obtain a marking quality score of the target task type executed by the candidate marking guest; and based on the annotation quality score of each candidate annotation customer, allocating a target annotation customer to each to-be-annotated file in the to-be-allocated annotation task. According to the scheme, the rationality of annotation task allocation can be improved.

Description

Technical Field

[0001] The present application relates to the field of data annotation technology, and in particular to a method for allocating annotation tasks and related devices, equipment and storage media. Background Art

[0002] With the development of artificial intelligence technology, supervised learning technology based on machine learning has been applied to more and more fields. In supervised learning, machine models are trained based on training data, so the task of labeling sample data has received more and more attention.

[0003] In the prior art, training data is usually manually labeled by labelers, and labeling tasks are usually randomly assigned to each labeler based on random distribution by managers or random collection by labelers. However, due to the inevitable subjective differences in cognitive abilities of managers and labelers, the rationality of labeling task allocation is reduced. In view of this, how to improve the rationality of labeling task allocation has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a labeling task allocation method and related devices, equipment and storage media, which can improve the rationality of labeling task allocation.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a labeling task assignment method, including: based on the target task type of the labeling task to be assigned, obtaining indicator data of the target task type performed by the candidate labelers; wherein the indicator data includes indicator values ​​of several indicator dimensions; based on the configuration information about the relative emphasis of several indicator dimensions under the target task type, obtaining the weight factor of each indicator dimension under the target task type; weighting based on the weight factor and indicator value of the same indicator dimension under the target task type to obtain the labeling quality score of the target task type performed by the candidate labelers; based on the labeling quality score of each candidate labeler, assigning a target labeler to each file to be labeled in the labeling task to be assigned.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a labeling task assignment device, including: an acquisition module, a determination module, a scoring module and an assignment module, the acquisition module is used to obtain the indicator data of the candidate labeler performing the target task type based on the target task type of the labeling task to be assigned; wherein the indicator data includes the indicator values ​​of several indicator dimensions; the determination module is used to obtain the weight factors of each indicator dimension under the target task type based on the configuration information about the relative emphasis of several indicator dimensions under the target task type; the scoring module is used to weight the weight factors and indicator values ​​of the same indicator dimensions under the target task type to obtain the labeling quality score of the candidate labeler performing the target task type; the assignment module is used to assign a target labeler to each file to be labeled in the labeling task to be assigned based on the labeling quality score of each candidate labeler.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, the memory storing program instructions, and the processor being used to execute the program instructions to implement the labeling task allocation method in the above first aspect.

[0008] In order to solve the above technical problem, the fourth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the labeling task allocation method described in the first aspect above.

[0009] The above scheme obtains the index data of each candidate bidder for executing the target task type based on the target task type of the labeling task to be assigned, and the index data includes the index values ​​of several index dimensions, obtains the weight factors of each index dimension under the target task type based on the configuration information of the relative emphasis of several index dimensions under the target task type, and weights the weight factors and index values ​​of the same index dimensions under the target task type to obtain the labeling quality score of the candidate bidder for executing the target task type, and assigns the target bidder to each file to be labeled in the labeling task to be assigned based on the labeling quality score of each candidate bidder. On the one hand, the index data of the candidate bidder obtained based on the task type of the labeling task to be assigned can provide as relevant reference information as possible for the assignment of the labeling task, and on the other hand, the corresponding weight factors are constructed for each index dimension under the target task type, and the labeling quality score of the candidate bidder obtained based on the weighting of the weight factors can reflect the labeling quality of each candidate bidder as objectively as possible while considering the relative emphasis of several index dimensions, and improve the quality and efficiency of task assignment when assigning each file to be labeled in the labeling task to be assigned. Therefore, the rationality of labeling task assignment can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of an embodiment of the method for allocating labeling tasks of the present application; Figure 2 It is a schematic diagram of obtaining the average accuracy rate in the labeling task allocation method of this application; Figure 3 It is a flowchart of another embodiment of the method for allocating labeling tasks of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of the labeling task allocation device of the present application; Figure 5 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 6 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0011] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0012] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0013] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0014] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the method for allocating labeling tasks in the present application. Specifically, it may include the following steps: Step S10: Based on the target task type of the labeling task to be assigned, obtain the indicator data of the target task type performed by the candidate labeler.

[0015] In the disclosed embodiment, there is no limitation on the task type of the labeling task to be assigned, such as image labeling, region labeling, relationship labeling, question and answer labeling, etc.

[0016] In an implementation scenario, the target task type is determined by a number of classification labels, and the classification labels of any two tasks of the same type are the same. For example, the classification labels include language dimension, technical point dimension and scenario dimension. The language dimension includes classification labels such as Chinese, English, and German, the technical point dimension includes classification labels such as dictation, transcription, and translation, and the scenario dimension includes classification labels such as conference, input method, and intelligent question and answer. Specifically, the target task type includes classification labels such as "Chinese, dictation, and conference".

[0017] In a specific implementation scenario, a candidate bidder processes historical annotation tasks of multiple task types. For example, the historical annotation tasks include task A, task B, task C, and task D. Taking the aforementioned embodiment as an example, the task type of task A is "Chinese, dictation, meeting", the task type of task B is "English, dictation, input method", the task type of task C is "English, transcription, input method", and the task type of task D is "Chinese, dictation, meeting". The target task type of the annotation task to be assigned is "Chinese, dictation, meeting", that is, the indicator data of the candidate bidder performing tasks A and D is obtained.

[0018] In a specific implementation scenario, before obtaining the indicator data of the target task type performed by the candidate labeler based on the target task type of the labeling task to be assigned, several tasks to be assigned are merged and screened based on the task type, so that any labeling task to be assigned contains several files to be labeled, and each file to be labeled belongs to the same target task type.

[0019] In a specific implementation scenario, when any file to be marked is marked by bidders in several rounds of marking steps in sequence, any bidder can only participate in one round of marking step in any file. For example, the file to be marked includes a marking step, a first round of inspection step and a second round of inspection step set in sequence, and candidate bidder A can only perform the marking task of any one of the marking step, the first round of inspection step and the second round of inspection step.

[0020] In another specific implementation scenario, when any file to be marked is marked by bidders in several rounds of marking steps in sequence, any bidder can participate in multiple non-adjacent rounds of marking steps in any file. For example, the file to be marked includes a marking step, a first round of inspection step, a second round of inspection step and a third round of inspection step set in sequence, and candidate bidder B can perform the marking tasks of the first round of inspection step and the third round of inspection step.

[0021] In an implementation scenario, the indicator data includes indicator values ​​of several indicator dimensions, and the indicator values ​​are used to quantify each indicator dimension, and quantify the annotation operation data used to characterize the annotation quality of the candidate labelers into specific and measurable values. In addition, the indicator data of the candidate labelers obtained based on the task type of the annotation task to be assigned can provide reference information that is as relevant as possible for the assignment of the annotation task.

[0022] In a specific implementation scenario, the indicator dimensions include at least one of: average accuracy, task abandonment rate, and average labeling time.

[0023] In a specific implementation scenario, the average accuracy represents the average value of the accuracy of the candidate labelers' annotations of historical annotation tasks under the target task type.

[0024] See also Figure 2 , Figure 2 Schematic diagram of the average accuracy rate obtained in the labeling task allocation method of this application. Figure 2 As shown, when any file is annotated by bidders in several rounds of annotation steps in sequence, for example, including a sequentially set annotation step, a first round of inspection step and a second round of inspection step, the annotation data of the candidate bidders participating in the target annotation task are obtained as the target annotation data, the target annotation task is a historical annotation task of the target task type, and the final annotation data of the last round of annotation in the target annotation task is obtained, and based on the similarity between the target annotation data and the final annotation data belonging to the same target annotation task, the indicator value of the average accuracy of the candidate bidder under the target task type is obtained.

[0025] In a specific implementation scenario, when any file to be annotated is annotated by bidders in several rounds of annotation stages in sequence, and any bidder can only participate in one round of annotation stages in any file, obtain the index value of the average accuracy rate of the candidate bidder under the target task type. For example, in the historical annotation task of the target task type, candidate bidder A participated in the annotation stage of historical annotation file 1 and one round of inspection stage of historical annotation file 2, and both historical annotation file 1 and historical annotation file 2 include the annotation stage, the first round of inspection stage, the second round of inspection stage and the third round of inspection stage set in sequence, that is, the third round of inspection stage is the last round of annotation stage in the target annotation task, and obtains Take the final annotation data of the three rounds of inspection in historical annotation file 1 and the final annotation data of the three rounds of inspection in historical annotation file 2, and obtain the first accuracy of candidate bidder A for historical annotation file 1 based on the similarity between the target annotation data of the annotation phase in historical annotation file 1 and the final annotation data of the three rounds of inspection, and obtain the second accuracy of candidate bidder A for historical annotation file 2 based on the similarity between the target annotation data of one round of inspection in historical annotation file 2 and the final annotation data of the three rounds of inspection, and obtain the index value of the average accuracy of candidate bidder A under the target task type based on the average of the first accuracy and the second accuracy.

[0026] In a specific implementation scenario, when any file to be annotated is annotated by bidders in several rounds of annotation in sequence, and any bidder can participate in multiple non-adjacent rounds of annotation in any file, taking the aforementioned embodiment as an example, candidate bidder B participated in one round of inspection of historically annotated file 1, the annotation of historically annotated file 2, and two rounds of inspection of historically annotated file 2. Based on the similarity between the target annotation data and the final annotation data in the same target annotation task, the index value of the average accuracy of candidate bidder B under the target task type is obtained.

[0027] In a specific implementation scenario, the method for obtaining the similarity between the target annotation data and the final annotation data is related to the task type of the target annotation task. For example, the target annotation task of the text annotation type can calculate the similarity based on the LD algorithm (edit distance algorithm, Levenshtein distance), and the target annotation task of the area selection type can calculate the similarity based on the overlapping area ratio algorithm. It should be noted that the above embodiment is only a possible implementation method, and the specific similarity calculation method is not limited in this application.

[0028] In a specific implementation scenario, the task abandonment rate is obtained based on the ratio between the number of abandonments and the number of assignments of the candidate bidder for historical labeling tasks under the target task type.

[0029] In a specific implementation scenario, the average labeling time is obtained based on the average value of the difference between the completion time and the allocation time of each historical labeling task of the candidate labeler under the target task type.

[0030] It should be noted that the number of indicator dimensions and the specific meanings of the dimensions represented are not limited in this application. For example, the indicator dimensions also include the proportion of annotation types, etc.

[0031] Step S20: Based on the configuration information of the relative emphasis of several indicator dimensions under the target task type, a weight factor of each indicator dimension under the target task type is obtained.

[0032] In an implementation scenario, the configuration information includes: the relative emphasis of any indicator dimension compared to each indicator dimension. For example, several indicator dimensions include: average accuracy, task abandonment rate, and average labeling time. Please refer to the following Table 1:

[0033] Table 1 As shown in Table 1, the relative emphasis value of the average accuracy rate compared to the task abandonment rate is 7, that is, the importance of the average accuracy rate to the annotation quality score is greater than the task abandonment rate. It should be noted that the specific values ​​of the relative emphasis values ​​of any indicator dimension compared to each indicator dimension are not limited in this application, and the specific values ​​are related to the importance of the indicator dimension to the annotation quality score.

[0034] In a specific implementation scenario, for any indicator dimension, based on the relative emphasis of the indicator dimension compared to each indicator dimension, the comprehensive value of the relative emphasis of the corresponding indicator dimension is obtained, and the comprehensive value of the relative emphasis of each indicator dimension is normalized to obtain the weight factor of each indicator dimension. The above scheme constructs corresponding weight factors for each indicator dimension under the target task type, and the annotation quality score of the candidate labeler obtained by weighting based on the weight factor can reflect the annotation quality of each candidate labeler as objectively as possible while considering the relative emphasis of several indicator dimensions, and improves the quality and efficiency of task allocation when allocating each to-be-annotated file in the to-be-assigned annotation task.

[0035] In a specific implementation scenario, the indicator dimension is multiplied by the relative emphasis of each indicator dimension to obtain the cumulative value, and the cumulative value corresponding to the indicator dimension is rooted based on the total number of several indicator dimensions to obtain the comprehensive value of the indicator dimension with respect to the relative emphasis, and the root index of the root operation is the total number. For example, still taking Table 1 as an example, the average labeling time = , task abandonment rate = , average accuracy = , the normalized weight factor corresponding to the average labeling time is calculated to be 0.8434 / (0.8434+0.3625+3.2711)=0.1884, the normalized weight factor corresponding to the task abandonment rate is 0.3625 / (0.8434+0.3625+3.2711)=0.081, and the normalized weight factor corresponding to the average accuracy is 3.2711 / (0.8434+0.3625+3.2711)=0.7306.

[0036] Step S30: weighting is performed based on the weight factors and indicator values ​​of the same indicator dimension under the target task type to obtain the annotation quality score of the target task type performed by the candidate labeler.

[0037] In an implementation scenario, weighting is performed based on the weight factors and indicator values ​​of the same indicator dimensions under the target task type to obtain the annotation quality score of the candidate bidder performing the target task type. On the one hand, the indicator data of the candidate bidder obtained based on the task type of the annotation task to be assigned can provide as relevant reference information as possible for the assignment of the annotation task. On the other hand, corresponding weight factors are constructed for each indicator dimension under the target task type. The annotation quality score of the candidate bidder obtained by weighting based on the weight factors can reflect the annotation quality of each candidate bidder as objectively as possible while considering the relative emphasis on several indicator dimensions. When allocating each file to be labeled in the annotation task to be assigned, the quality and efficiency of task assignment can be improved.

[0038] In an implementation scenario, the indicator values ​​of each candidate bidder on the same indicator dimension under the target task type are sorted from best to worst, and the scores of each candidate bidder on the corresponding indicator dimension under the target task type are obtained from high to low. The weighted values ​​of the same indicator dimension under the target task type are weighted to obtain the annotation quality score of the candidate bidder performing the target task type.

[0039] In a specific implementation scenario, the following Table 2 may be referred to:

[0040] Table 2 As shown in Table 2, including the indicator data characterizing the target task type performed by candidate bidder A, the indicator data characterizing the target task type performed by candidate bidder B, and the indicator data characterizing the target task type performed by candidate bidder C, it can be obtained that the physical meanings represented by the unit values ​​of the indicator values ​​under different indicator dimensions are different. Therefore, based on the indicator values ​​of each candidate bidder on the same indicator dimension under the target task type, they are arranged from best to worst, and the score values ​​of each candidate bidder on the corresponding indicator dimension under the target task type are obtained from high to low, as shown in the following Table 3:

[0041] Table 3 As shown in Table 3, the longer the average labeling time is, the worse the representation is, and the smaller the corresponding score is. The larger the value of the task abandonment rate is, the worse the representation is, and the smaller the corresponding score is. The smaller the value of the average accuracy is, the worse the representation is, and the smaller the corresponding score is. The above scheme reduces the differences in the meaning of numerical representation under different indicator dimensions, so it can improve the rationality of labeling task allocation.

[0042] Step S40: assigning a target labeler to each file to be labeled in the assigned labeling task based on the labeling quality score of each candidate labeler.

[0043] In an implementation scenario, based on the annotation quality scores of each candidate bidder, target bidders are assigned to each file to be annotated in the assigned annotation task. On the one hand, the indicator data of the candidate bidders obtained based on the task type of the assigned annotation task can provide as relevant reference information as possible for the assignment of the annotation task. On the other hand, corresponding weight factors are constructed for each indicator dimension under the target task type. The annotation quality scores of the candidate bidders obtained by weighting based on the weight factors can reflect the annotation quality of each candidate bidder as objectively as possible while considering the relative emphasis on several indicator dimensions. When allocating each file to be annotated in the assigned annotation task, the quality and efficiency of task assignment are improved. Therefore, the rationality of the assignment of annotation tasks can be improved.

[0044] In an implementation scenario, any file is annotated by bidders in several rounds of annotation, and candidate bidders are sorted in order of annotation quality scores from low to high to obtain a bidder sequence. Based on the total number of annotation stages, the bidder sequence is divided into several subsequences, and candidate bidders in each subsequence are selected in turn and assigned to each round of annotation from the front to the back. For example, the bidder sequence obtained by sorting each candidate bidder in order of the annotation quality score from low to high is {candidate bidder A, candidate bidder B, candidate bidder C, candidate bidder D, candidate bidder E, candidate bidder F, candidate bidder G, candidate bidder H, candidate bidder I}. The total number of annotation links of any file to be annotated is 3, which are the annotation link, the first round of inspection link and the second round of inspection link executed in sequence. The bidder sequence is divided into the first subsequence {candidate bidder A, candidate bidder B, candidate bidder C}, the second subsequence {candidate bidder D, candidate bidder E, candidate bidder F}, and the third subsequence {candidate bidder G, candidate bidder H, candidate bidder I}. Candidate bidder A is assigned to the annotation link of file 1 to be annotated, candidate bidder B is assigned to the annotation link of file 2 to be annotated, and candidate bidder D is assigned to the first round of inspection link of file 1 to be annotated, and so on. Therefore, the rationality of the annotation task allocation can be further improved.

[0045] In another implementation scenario, any file is annotated by a bidder in only one round of annotation, and the attribute information of each file to be annotated in the assigned annotation task is obtained, and the attribute information includes at least one of the importance and difficulty of the file to be annotated. Based on the attribute information of each file to be annotated and the annotation quality score of each candidate bidder, a target bidder is assigned to each file to be annotated in the assigned annotation task. For example, based on the attribute information of the file to be annotated, each file to be annotated in the assigned annotation task is sorted, and the sorting characterizes the files to be annotated from difficult to easy, and / or, from important to secondary, to obtain {file to be annotated 1, file to be annotated 2, file to be annotated 3}, and based on the annotation quality score of each candidate bidder, each candidate bidder is sorted in order from high to low, to obtain {candidate bidder 1, candidate bidder 2, candidate bidder 3, candidate bidder 4, candidate bidder 5}, therefore, file to be annotated 1 is assigned to candidate bidder 1, file to be annotated 2 is assigned to candidate bidder 2, and file to be annotated 3 is assigned to candidate bidder 3. Therefore, the rationality of the annotation task allocation can be further improved.

[0046] In an implementation scenario, any file is annotated by bidders in several rounds of annotation links in sequence, and each candidate bidder is annotated with a grade label, for example, the grade label includes grade A, grade B, grade C and grade E from high to low, and the grade labels of each candidate bidder are divided to obtain a set of bidders belonging to each grade label, and the rounds of annotation links from the back to the front are sequentially used as target annotation links, and based on the mapping relationship between the number of rounds of the target annotation link and the grade label, at least one bidder set corresponding to the target annotation link is selected as the target bidder set. For example, the file to be annotated includes the annotation links set in sequence. , the first round of inspection link, the second round of inspection link. When the second round of inspection link is used as the target labeling link, the target labeler set includes all candidate labelers of level A and level B. When the first round of inspection link is used as the target labeling link, the target labeler set includes all candidate labelers of level B and level C. When the labeling link is used as the target labeling link, the target labeler set includes all candidate labelers of level E. The candidate labelers in the target labeler set are sorted in descending order of the labeling quality score to obtain the target labeler sequence of the target labeler set. The candidate labeler ranked first in the target labeler sequence is selected and assigned to the target labeling link. For example, when the second round of inspection is used as the target annotation link, the target bidder sequence is {candidate bidder 1 (level A), candidate bidder 2 (level A), candidate bidder 3 (level B), candidate bidder 4 (level A), candidate bidder 5 (level B)}, and the second round of inspection in the to-be-assigned file is assigned to candidate bidder 1 of level A. When the first round of inspection is used as the target annotation link, the target bidder sequence is {candidate bidder 3 (level B), candidate bidder 6 (level C), candidate bidder 5 (level B)}, and the first round of inspection in the to-be-assigned file is assigned to candidate bidder 3 of level B, and so on. Therefore, the rationality of the annotation task allocation can be further improved.

[0047] The above scheme obtains the index data of each candidate bidder for executing the target task type based on the target task type of the labeling task to be assigned, and the index data includes the index values ​​of several index dimensions, obtains the weight factors of each index dimension under the target task type based on the configuration information of the relative emphasis of several index dimensions under the target task type, and weights the weight factors and index values ​​of the same index dimensions under the target task type to obtain the labeling quality score of the candidate bidder for executing the target task type, and assigns the target bidder to each file to be labeled in the labeling task to be assigned based on the labeling quality score of each candidate bidder. On the one hand, the index data of the candidate bidder obtained based on the task type of the labeling task to be assigned can provide as relevant reference information as possible for the assignment of the labeling task, and on the other hand, the corresponding weight factors are constructed for each index dimension under the target task type, and the labeling quality score of the candidate bidder obtained based on the weighting of the weight factors can reflect the labeling quality of each candidate bidder as objectively as possible while considering the relative emphasis of several index dimensions, and improve the quality and efficiency of task assignment when assigning each file to be labeled in the labeling task to be assigned. Therefore, the rationality of labeling task assignment can be improved.

[0048] See also Figure 3 , Figure 3 This is a flowchart of another embodiment of the labeling task allocation method of the present application. Several indicator dimensions include: average accuracy, task abandonment rate and average labeling time. Specifically, the following steps may be included: Step S11: Obtain the target task type of the labeling task to be assigned.

[0049] For details, please refer to the detailed description of the aforementioned embodiment, which will not be repeated here for the sake of brevity.

[0050] Step S111: Obtain the index value of the candidate bidder in performing the target task type under the average accuracy index dimension.

[0051] For details, please refer to the detailed description of the aforementioned embodiment, which will not be repeated here for the sake of brevity.

[0052] Step S112: Obtain the index value of the candidate bidder for executing the target task type under the task abandonment rate index dimension.

[0053] For details, please refer to the detailed description of the aforementioned embodiment, which will not be repeated here for the sake of brevity.

[0054] Step S113: Obtain the index value of the candidate labeler performing the target task type under the average labeling time index dimension.

[0055] For details, please refer to the detailed description of the aforementioned embodiment, which will not be repeated here for the sake of brevity.

[0056] Step S211: Based on the configuration information of the relative emphasis of several indicator dimensions under the target task type, a weight factor of the average accuracy indicator dimension under the target task type is obtained.

[0057] For details, please refer to the detailed description of the aforementioned embodiment, which will not be repeated here for the sake of brevity.

[0058] Step S212: Based on the configuration information of the relative emphasis of several indicator dimensions under the target task type, a weight factor of the task abandonment rate indicator dimension under the target task type is obtained.

[0059] For details, please refer to the detailed description of the aforementioned embodiment, which will not be repeated here for the sake of brevity.

[0060] Step S213: Based on the configuration information of the relative emphasis of several indicator dimensions under the target task type, a weight factor of the average labeling time-consuming indicator dimension under the target task type is obtained.

[0061] For details, please refer to the detailed description of the aforementioned embodiment, which will not be repeated here for the sake of brevity.

[0062] Step S30: weighting is performed based on the weight factors and indicator values ​​of the same indicator dimension under the target task type to obtain the annotation quality score of the target task type performed by the candidate labeler.

[0063] For details, please refer to the detailed description of the aforementioned embodiment, which will not be repeated here for the sake of brevity.

[0064] Step S40: assigning a target labeler to each file to be labeled in the assigned labeling task based on the labeling quality score of each candidate labeler.

[0065] For details, please refer to the detailed description of the aforementioned embodiment, which will not be repeated here for the sake of brevity.

[0066] The above scheme obtains the index data of each candidate bidder for executing the target task type based on the target task type of the labeling task to be assigned, and the index data includes the index values ​​of several index dimensions, obtains the weight factors of each index dimension under the target task type based on the configuration information of the relative emphasis of several index dimensions under the target task type, and weights the weight factors and index values ​​of the same index dimensions under the target task type to obtain the labeling quality score of the candidate bidder for executing the target task type, and assigns the target bidder to each file to be labeled in the labeling task to be assigned based on the labeling quality score of each candidate bidder. On the one hand, the index data of the candidate bidder obtained based on the task type of the labeling task to be assigned can provide as relevant reference information as possible for the assignment of the labeling task, and on the other hand, the corresponding weight factors are constructed for each index dimension under the target task type, and the labeling quality score of the candidate bidder obtained based on the weighting of the weight factors can reflect the labeling quality of each candidate bidder as objectively as possible while considering the relative emphasis of several index dimensions, and improve the quality and efficiency of task assignment when assigning each file to be labeled in the labeling task to be assigned. Therefore, the rationality of labeling task assignment can be improved.

[0067] See also Figure 4 , Figure 4 4 is a schematic diagram of a framework of an embodiment of a labeling task allocation device 40 of the present application. Figure 4 As shown, the labeling task allocation device 40 includes: an acquisition module 41, a determination module 42, a scoring module 43 and an allocation module 44, wherein the acquisition module 41 is used to acquire the indicator data of the target task type performed by the candidate labeler based on the target task type of the labeling task to be assigned; wherein the indicator data includes the indicator values ​​of several indicator dimensions; the determination module 42 is used to obtain the weight factor of each indicator dimension under the target task type based on the configuration information about the relative emphasis of several indicator dimensions under the target task type; the scoring module 43 is used to weight the weight factor and indicator value of the same indicator dimension under the target task type to obtain the labeling quality score of the candidate labeler performing the target task type; the allocation module 44 is used to allocate the target labeler to each file to be labeled in the labeling task to be assigned based on the labeling quality score of each candidate labeler.

[0068] In the above scheme, the labeling task allocation device 40 obtains the index data of each candidate bidder for executing the target task type based on the target task type of the labeling task to be allocated, and the index data includes the index values ​​of several index dimensions, obtains the weight factors of each index dimension under the target task type based on the configuration information of the relative emphasis of several index dimensions under the target task type, and weights the weight factors and index values ​​of the same index dimensions under the target task type to obtain the labeling quality score of the candidate bidder for executing the target task type, and allocates the target bidder to each to-be-annotated file in the labeling task to be allocated based on the labeling quality score of each candidate bidder. On the one hand, the index data of the candidate bidder obtained based on the task type of the labeling task to be allocated can provide as relevant reference information as possible for the allocation of the labeling task, and on the other hand, the corresponding weight factors are constructed for each index dimension under the target task type, and the labeling quality score of the candidate bidder obtained based on the weighting of the weight factors can reflect the labeling quality of each candidate bidder as objectively as possible under the condition of considering the relative emphasis of several index dimensions, and improve the quality and efficiency of task allocation when allocating each to-be-annotated file in the labeling task to be allocated. Therefore, the rationality of labeling task allocation can be improved.

[0069] In some disclosed embodiments, any file is labeled by bidders in several rounds of labeling in sequence. When the several indicator dimensions include at least the average accuracy, the acquisition module 41 also includes a labeling data acquisition module (not shown) for acquiring the labeling data of candidate bidders participating in the target labeling task as the target labeling data, and acquiring the final labeling data of the last round of labeling in the target labeling task; wherein the target labeling task is a historical labeling task of the target task type; the acquisition module 41 also includes a similarity comparison module (not shown) for obtaining the indicator value of the candidate bidder regarding the average accuracy under the target task type based on the similarity between the target labeling data and the final labeling data belonging to the same target labeling task.

[0070] In some disclosed embodiments, the scoring module 43 also includes a scoring module (not shown), which is used to sort the indicator values ​​of each candidate bidder for the same indicator dimension under the target task type in order from best to worst, and obtain the score values ​​of each candidate bidder for the corresponding indicator dimension under the target task type from high to low; the scoring module 43 also includes a weighting module (not shown), which is used to weight based on the weight factor and the score value of the same indicator dimension under the target task type, and obtain the annotation quality score of the candidate bidder performing the target task type.

[0071] In some disclosed embodiments, any file is annotated by bidders in several rounds of annotation steps in sequence, and the allocation module 44 further includes a sorting module (not shown) for sorting the candidate bidders in order of the annotation quality score from low to high to obtain a bidder sequence; the allocation module 44 further includes a division module (not shown) for dividing the bidder sequence into several subsequences based on the total number of annotation steps; the allocation module 44 further includes a selection module (not shown) for selecting candidate bidders in each subsequence in sequence and assigning them to each round of annotation steps from the front to the back.

[0072] In some disclosed embodiments, any file is annotated by bidders in only one round of annotation, and the assignment module 44 further includes an attribute acquisition module (not shown) for acquiring attribute information of each file to be annotated in the annotation task to be assigned; wherein the attribute information includes at least one of the importance and difficulty of the file to be annotated; the assignment module 44 further includes a score assignment module (not shown) for assigning a target bidder to each file to be annotated in the annotation task to be assigned based on the attribute information of each file to be annotated and the annotation quality score of each candidate bidder.

[0073] In some disclosed embodiments, the configuration information includes: the degree of relative emphasis of any indicator dimension compared to each indicator dimension, the determination module 42 also includes a comprehensive value acquisition module (not shown), which is used to obtain the comprehensive value of relative emphasis of the corresponding indicator dimension for any indicator dimension based on the degree of relative emphasis of the indicator dimension compared to each indicator dimension; the determination module 42 also includes a normalization module (not shown), which is used to normalize based on the comprehensive value of relative emphasis of each indicator dimension to obtain the weight factor of each indicator dimension.

[0074] In some disclosed embodiments, the comprehensive value acquisition module (not shown) also includes a multiplication module (not shown) for performing multiplication based on the indicator dimension compared to the degree of relative emphasis of each indicator dimension to obtain a cumulative value; the comprehensive value acquisition module (not shown) also includes a root operation module (not shown) for performing a root operation on the cumulative value corresponding to the indicator dimension based on the total number of several indicator dimensions to obtain a comprehensive value of the indicator dimension with respect to relative emphasis; wherein the root index of the root operation is the total number.

[0075] In some disclosed embodiments, any file is annotated by bidders in several rounds of annotation links in sequence, and each candidate bidder is annotated with a grade label. The allocation module 44 (not shown) further includes a grade division module (not shown) for dividing based on the grade labels of each candidate bidder to obtain a set of bidders belonging to each grade label, and sequentially taking each round of annotation links from the back to the front as the target annotation link; the allocation module 44 (not shown) further includes a set selection module (not shown) for selecting at least one bidder set corresponding to the target annotation link as the target bidder set based on the mapping relationship between the number of rounds of the target annotation link and the grade label; the allocation module 44 (not shown) further includes a sequence acquisition module (not shown) for sorting each candidate bidder in the target bidder set in descending order of the annotation quality score to obtain a target bidder sequence of the target bidder set; the allocation module 44 (not shown) further includes a bidder selection module (not shown) for selecting the candidate bidder ranked first in the target bidder sequence and assigning it to the target annotation link.

[0076] In some disclosed embodiments, several indicator dimensions in the labeling task allocation device 40 include: average accuracy, task abandonment rate, average labeling time; and / or, the target task type is determined by several classification labels, and the classification labels of any two tasks of the same type are the same.

[0077] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an embodiment of an electronic device 50 of the present application. The electronic device 50 includes a memory 51 and a processor 52. The memory 51 stores program instructions, and the processor 52 is used to execute the program instructions to implement the steps in any of the above-mentioned embodiments of the labeling task allocation method. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. The electronic device 50 may specifically include but is not limited to: a server, a smart phone, a laptop computer, a tablet computer, a self-service machine, etc., which are not limited here.

[0078] Specifically, the processor 52 is used to control itself and the memory 51 to implement the steps in any of the above-mentioned labeling task allocation method embodiments. The processor 52 can also be called a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 52 can be implemented by an integrated circuit chip.

[0079] In the above scheme, the electronic device 50 obtains the index data of each candidate bidder for executing the target task type based on the target task type of the labeling task to be assigned, and the index data includes the index values ​​of several index dimensions, obtains the weight factors of each index dimension under the target task type based on the configuration information of the several index dimensions under the target task type about the relative emphasis, and weights the weight factors and index values ​​of the same index dimensions under the target task type to obtain the labeling quality score of the candidate bidder for executing the target task type, and assigns the target bidder to each file to be labeled in the labeling task to be assigned based on the labeling quality score of each candidate bidder. On the one hand, the index data of the candidate bidder obtained based on the task type of the labeling task to be assigned can provide as relevant reference information as possible for the assignment of the labeling task, and on the other hand, the corresponding weight factors are constructed for each index dimension under the target task type, and the labeling quality score of the candidate bidder obtained by weighting based on the weight factors can reflect the labeling quality of each candidate bidder as objectively as possible under the condition of considering the relative emphasis of several index dimensions, and improve the quality and efficiency of task assignment when assigning each file to be labeled in the labeling task to be assigned. Therefore, the rationality of labeling task assignment can be improved.

[0080] See also Figure 6 , Figure 6 The figure is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 60 of the present application. The computer-readable storage medium 60 stores program instructions 61 that can be executed by a processor, and the program instructions 61 are used to implement the steps in any of the above-mentioned embodiments of the labeling task allocation method.

[0081] In the above scheme, the computer-readable storage medium 60 obtains the index data of each candidate bidder for executing the target task type based on the target task type of the labeling task to be assigned, and the index data includes the index values ​​of several index dimensions, obtains the weight factors of each index dimension under the target task type based on the configuration information of the relative emphasis of several index dimensions under the target task type, and weights the weight factors and index values ​​of the same index dimensions under the target task type to obtain the labeling quality score of the candidate bidder for executing the target task type, and assigns the target bidder to each file to be labeled in the labeling task to be assigned based on the labeling quality score of each candidate bidder. On the one hand, the index data of the candidate bidder obtained based on the task type of the labeling task to be assigned can provide as relevant reference information as possible for the assignment of the labeling task, and on the other hand, the corresponding weight factors are constructed for each index dimension under the target task type, and the labeling quality score of the candidate bidder obtained based on the weighting of the weight factors can reflect the labeling quality of each candidate bidder as objectively as possible under the condition of considering the relative emphasis of several index dimensions, and improve the quality and efficiency of task assignment when assigning each file to be labeled in the labeling task to be assigned. Therefore, the rationality of labeling task assignment can be improved.

[0082] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0083] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0084] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0085] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0086] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0087] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0088] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A method for allocating labeling tasks, characterized in that: include: Based on the target task type of the labeling task to be assigned, obtaining indicator data of the candidate labeler performing the target task type; wherein the indicator data includes indicator values ​​of several indicator dimensions; Based on the configuration information about the relative emphasis of the several indicator dimensions under the target task type, a weight factor of each indicator dimension under the target task type is obtained; Weighting is performed based on the weight factor and the indicator value of the same indicator dimension under the target task type to obtain the annotation quality score of the candidate labeler performing the target task type; Based on the annotation quality scores of the candidate bidders, target bidders are assigned to the files to be annotated in the annotation tasks to be assigned.

2. The method according to claim 1, characterized in that Any document is annotated by bidders in several rounds of annotation. When the several indicator dimensions include at least the average accuracy, the indicator data of the target task type performed by the candidate bidders based on the target task type to be assigned annotation tasks is obtained, including: Acquire the labeling data of the target labeling task participated by the candidate bidder as the target labeling data, and acquire the final labeling data of the last round of labeling in the target labeling task; wherein the target labeling task is a historical labeling task of the target task type; Based on the similarity between the target annotation data and the final annotation data belonging to the same target annotation task, an index value of the candidate labeler regarding the average accuracy under the target task type is obtained.

3. The method according to claim 1, characterized in that The weighting of the weight factors and indicator values ​​of the same indicator dimension under the target task type to obtain the annotation quality score of the candidate labeler performing the target task type includes: Based on the index values ​​of the candidate bidders for the same index dimension under the target task type, the index values ​​of the candidate bidders are sorted from best to worst, and the scoring values ​​of the candidate bidders for the corresponding index dimension under the target task type are obtained from high to low; Weighting is performed based on the weight factors and score values ​​of the same indicator dimensions under the target task type to obtain a labeling quality score of the candidate labeler performing the target task type.

4. The method according to claim 1, characterized in that: Any file is labeled by bidders in several rounds of labeling. The target bidders are assigned to each file to be labeled in the labeling task based on the labeling quality scores of each candidate bidder, including: Sorting the candidate bidders in order of the annotation quality scores from low to high to obtain a bidder sequence; Based on the total number of the labeling links, dividing the bidder sequence into a plurality of subsequences; The candidate labelers in each of the subsequences are selected in turn and assigned to each round of the labeling process from the front to the back.

5. The method according to claim 1, characterized in that Any file is annotated by bidders only in one round of annotation. The method of assigning a target bidder to each file to be annotated in the assigned annotation task based on the annotation quality score of each candidate bidder includes: Acquire attribute information of each of the to-be-annotated files in the to-be-assigned annotation task; wherein the attribute information includes at least one of the importance and difficulty of the to-be-annotated files; Based on the attribute information of each of the to-be-annotated files and the annotation quality score of each of the candidate bidders, the target bidder is assigned to each of the to-be-annotated files in the to-be-assigned annotation task.

6. The method according to claim 1, characterized in that The configuration information includes: the relative emphasis of any indicator dimension compared to each indicator dimension, and the configuration information about the relative emphasis of the several indicator dimensions under the target task type is used to obtain the weight factor of each indicator dimension under the target task type, including: For any of the indicator dimensions, based on the relative emphasis values ​​of the indicator dimension compared to each of the indicator dimensions, a comprehensive value of the relative emphasis of the corresponding indicator dimension is obtained; Based on the comprehensive values ​​of the relative emphasis of each indicator dimension, normalization is performed to obtain the weight factor of each indicator dimension.

7. The method according to claim 6, characterized in that The step of obtaining a comprehensive value of the relative emphasis of the corresponding indicator dimension based on the indicator dimension compared with the relative emphasis degree value of each indicator dimension includes: Based on the indicator dimension, the relative emphasis degree values ​​of each indicator dimension are respectively multiplied to obtain a cumulative value; Based on the total number of the several indicator dimensions, a root operation is performed on the cumulative multiplication value corresponding to the indicator dimension to obtain a comprehensive value of the indicator dimension with respect to relative emphasis; wherein the root index of the root operation is the total number.

8. The method according to claim 1, characterized in that Any file is annotated by bidders in several rounds of annotation, and each candidate bidder is annotated with a grade label. The target bidder assigned to each file to be annotated in the assigned annotation task based on the annotation quality score of each candidate bidder includes: Based on the level labels of the candidate bidders, the bidder sets belonging to the level labels are obtained, and the labeling stages from the last to the front are sequentially used as target labeling stages; Based on the mapping relationship between the round number of the target labeling stage and the level label, selecting at least one of the bidder sets corresponding to the target labeling stage as the target bidder set; Sorting the candidate bidders in descending order of the annotation quality scores in the target bidder set to obtain a target bidder sequence of the target bidder set; The candidate bidder ranked first in the target bidder sequence is selected and assigned to the target labeling stage.

9. The method according to any one of claims 1 to 8, characterized in that: The several indicator dimensions include: average accuracy, task abandonment rate, and average labeling time; And / or, the target task type is determined by a plurality of classification labels, and the classification labels of any two tasks of the same type are the same.

10. A labeling task allocation device, characterized in that: include: An acquisition module, used to acquire, based on the target task type of the labeling task to be assigned, the index data of the candidate labeler performing the target task type; wherein the index data includes the index values ​​of several indicator dimensions; A determination module, configured to obtain a weight factor of each of the indicator dimensions under the target task type based on the configuration information of the relative emphasis of the several indicator dimensions under the target task type; A scoring module, used for weighting based on the weight factor and the indicator value of the same indicator dimension under the target task type to obtain the annotation quality score of the candidate labeler performing the target task type; The allocation module is used to allocate a target labeler to each file to be labeled in the labeling task to be allocated based on the labeling quality score of each candidate labeler.

11. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the labeling task allocation method according to any one of claims 1 to 9.

12. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the labeling task allocation method according to any one of claims 1 to 9 is implemented.

Images