Method, device and equipment for distributing tasks to be labeled and storage medium

Abstract

The invention provides a to-be-labeled task allocation method and device, equipment and a storage medium, and the method enables a task executor with high data labeling accuracy to be able to allocate more to-be-labeled questions with higher labeling difficulty by improving the allocation mode of to-be-labeled questions. A task executor with low data annotation accuracy can distribute to-be-annotated questions with relatively small quantity and lower annotation difficulty, so that the annotation times required by answer distribution convergence of the to-be-annotated questions can be reduced; therefore, the overall labeling efficiency of the to-be-labeled task is improved, and meanwhile the overall labeling cost of the to-be-labeled task can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the technical field of data annotation, and in particular to a method, device, equipment and storage medium for allocating tasks to be labeled. Background Art

[0002] The labeling workload of data labeling tasks is large and tedious, and is usually achieved through crowdsourcing labeling by non-experts. The task initiator can publish the data labeling task through the crowdsourcing platform. The crowdsourcing platform can distribute the unlabeled topics in the data labeling task to multiple different labelers for labeling, thereby helping the task initiator to recover massive amounts of labeled data.

[0003] In the existing allocation scheme, the crowdsourcing platform randomly selects an equal number of questions to be labeled from multiple questions to be labeled and allocates them to different annotators; however, this equal random allocation method does not take into account the different labeling difficulties of different questions to be labeled and the different data labeling accuracy of different annotators, thus failing to maximize the value of the annotator group, leaving room for optimization in the allocation of questions to be labeled. Summary of the invention

[0004] In view of this, the present application provides a method, device, equipment and storage medium for allocating tasks to be labeled. By improving the allocation method of questions to be labeled, task executors with high data labeling accuracy can be allocated more questions to be labeled with higher labeling difficulty, while task executors with low data labeling accuracy can be allocated relatively fewer questions to be labeled with lower labeling difficulty. This is beneficial to reducing the number of labeling times required for the convergence of the answer distribution of the questions to be labeled, thereby improving the overall labeling efficiency of the tasks to be labeled and effectively reducing the overall labeling cost of the tasks to be labeled.

[0005] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings.

[0006] In a first aspect, an embodiment of the present application provides a method for allocating tasks to be labeled, the method comprising:

[0007] In response to a task assignment request of a task to be labeled, according to the data labeling accuracy of a target task executor, the number of assignable questions matching the data labeling accuracy is determined; wherein the correlation direction between the data labeling accuracy and the number of assignable questions is positively correlated;

[0008] If the data annotation accuracy is less than or equal to a preset accuracy threshold, obtaining the first to-be-annotated topic of the allocatable number of topics from the multiple to-be-annotated topics constituting the to-be-annotated task and assigning them to the target task executor;

[0009] If the data annotation accuracy is greater than a preset accuracy threshold, a second set of topics to be labeled with the number of assignable topics is obtained from a plurality of topics to be labeled constituting the task to be labeled and assigned to the target task executor; wherein the annotation difficulty of the second topic to be labeled is greater than that of the first topic to be labeled.

[0010] In a second aspect, an embodiment of the present application provides a device for allocating tasks to be labeled, the device comprising:

[0011] A response module, for responding to a task assignment request of a task to be labeled, and determining, according to the data labeling accuracy of a target task executor, the number of assignable questions that matches the data labeling accuracy; wherein the correlation direction between the data labeling accuracy and the number of assignable questions is positively correlated;

[0012] A first allocation module is configured to obtain the first number of allocable topics to be labeled from the multiple topics to be labeled constituting the task to be labeled and allocate them to the target task executor if the data labeling accuracy is less than or equal to a preset accuracy threshold;

[0013] A second allocation module is used to obtain second questions to be labeled with the number of allocable questions from multiple questions to be labeled that constitute the task to be labeled and allocate them to the target task executor if the data labeling accuracy is greater than a preset accuracy threshold; wherein the labeling difficulty of the second questions to be labeled is greater than that of the first questions to be labeled.

[0014] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for allocating tasks to be labeled when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for allocating tasks to be labeled are executed.

[0016] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0017] The embodiments of the present application provide a method, device, equipment and storage medium for allocating tasks to be labeled. By improving the allocation method of questions to be labeled, task executors with high data labeling accuracy can be allocated more questions to be labeled with higher labeling difficulty, while task executors with low data labeling accuracy can be allocated relatively fewer questions to be labeled with lower labeling difficulty. This is beneficial to reducing the number of labeling times required for the convergence of the answer distribution of the questions to be labeled, thereby improving the overall labeling efficiency of the tasks to be labeled and effectively reducing the overall labeling cost of the tasks to be labeled. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic diagram of a flow chart of a method for allocating tasks to be labeled provided in an embodiment of the present application is shown;

[0020] Figure 2 A flow chart of a method for determining the number of allocable questions that matches the data annotation accuracy provided by an embodiment of the present application is shown;

[0021] Figure 3 A flow chart of a method for allocating a second to-be-annotated topic to a target task executor provided in an embodiment of the present application is shown;

[0022] Figure 4 A schematic diagram of a process for timely updating the marking difficulty of a to-be-marked question provided by an embodiment of the present application is shown;

[0023] Figure 5 A schematic diagram of the structure of a device for allocating tasks to be labeled provided in an embodiment of the present application is shown;

[0024] Figure 6 A schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0026] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0027] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0028] In the existing allocation scheme, the crowdsourcing platform randomly selects an equal number of questions to be labeled from multiple questions to be labeled and allocates them to different annotators; however, this equal random allocation method does not take into account the different labeling difficulties of different questions to be labeled and the different data labeling accuracy of different annotators, thus failing to maximize the value of the annotator group, leaving room for optimization in the allocation of questions to be labeled.

[0029] Based on this, the embodiments of the present application provide a method, device, equipment and storage medium for allocating tasks to be labeled. By improving the allocation method of questions to be labeled, task executors with high data labeling accuracy can be allocated more questions to be labeled with higher labeling difficulty, while task executors with low data labeling accuracy can be allocated relatively fewer questions to be labeled with lower labeling difficulty. This is conducive to reducing the number of labeling times required for the convergence of the answer distribution of the questions to be labeled, thereby improving the overall labeling efficiency of the tasks to be labeled while effectively reducing the overall labeling cost of the tasks to be labeled.

[0030] In one embodiment of the present application, a method for allocating tasks to be labeled can be run on a terminal device or a server. The terminal device can be a local terminal device. When the method for allocating tasks to be labeled is run on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device (i.e., a terminal device).

[0031] A method for allocating tasks to be labeled in one of the embodiments of the present application can also be run in various distribution systems involving data labeling, wherein the distribution system can be a distribution system in a crowdsourcing platform.

[0032] To facilitate understanding of the embodiments of the present application, a method, device, equipment and storage medium for allocating tasks to be labeled provided in the embodiments of the present application are described in detail below.

[0033] Reference Figure 1 As shown, Figure 1 A schematic flow chart of a method for assigning tasks to be labeled provided in an embodiment of the present application is shown, wherein the method includes steps S101-S103; specifically:

[0034] S101, in response to a task assignment request for a task to be labeled, determining the number of assignable topics that match the data labeling accuracy according to the data labeling accuracy of a target task executor.

[0035] S102: If the data annotation accuracy is less than or equal to a preset accuracy threshold, obtain the first number of assignable topics from the multiple topics to be labeled constituting the task to be labeled and assign them to the target task executor.

[0036] S103, if the data annotation accuracy is greater than a preset accuracy threshold, obtaining a second number of topics to be labeled that are assignable from a plurality of topics to be labeled constituting the task to be labeled and assigning them to the target task executor.

[0037] The above-mentioned method for allocating tasks to be labeled provided in the embodiment of the present application improves the allocation method of questions to be labeled, so that task executors with high data labeling accuracy can be allocated more questions to be labeled with higher labeling difficulty, while task executors with low data labeling accuracy can be allocated relatively fewer questions to be labeled with lower labeling difficulty, which is beneficial to reducing the number of labeling times required for the convergence of the answer distribution of the questions to be labeled, thereby improving the overall labeling efficiency of the tasks to be labeled and effectively reducing the overall labeling cost of the tasks to be labeled.

[0038] The following takes the allocation system applied to the crowdsourcing platform as an example to exemplify each step in the above-mentioned method for allocating tasks to be labeled provided in the embodiment of the present application:

[0039] S101, in response to a task assignment request for a task to be labeled, determining the number of assignable topics that match the data labeling accuracy according to the data labeling accuracy of a target task executor.

[0040] In an embodiment of the present application, the task to be labeled represents a data labeling task to be assigned in the crowdsourcing platform; wherein, as an optional embodiment, the crowdsourcing platform may be responsive to a task assignment request initiated by a labeler (i.e., the target task executor) for the task to be labeled, and screen out the topics to be labeled that need to be assigned to the labeler for labeling from the multiple topics to be labeled that constitute the task to be labeled; as another optional embodiment, the crowdsourcing platform may also recognize that the task to be labeled belongs to a data labeling task that needs to be assigned (equivalent to the fact that the data labeling results for each topic to be labeled in the task to be labeled have not yet been finally determined, and still need to be assigned to the labeler for labeling), and actively initiate a task assignment request for the task to be labeled (equivalent to automatically determining that the task assignment request for the task to be labeled needs to be responded to after recognizing that the task to be labeled belongs to a data labeling task that needs to be assigned); the embodiment of the present application does not impose any limitation on the specific method of initiating the above-mentioned task assignment request.

[0041] Here, the target task executor can represent any labeler who can perform the data labeling task (that is, the above-mentioned task to be labeled) and his / her corresponding terminal device. That is, taking the allocation method provided in this application as an example of an allocation system in a crowdsourcing platform, if the crowdsourcing platform does not limit the labeler's task acceptance authority (equivalent to the labeler being able to freely accept any type of data labeling task regardless of whether he / she has experience in data labeling tasks), then the target task executor can represent any labeler who can perform the data labeling task and his / her corresponding terminal device.

[0042] Here, considering that the fields (i.e., the types) to which the data to be labeled (i.e., the topics to be labeled in subsequent steps S102-S103) belong in different data labeling tasks (i.e., the above-mentioned tasks to be labeled) may differ (for example, when labeling image data, sometimes medical imaging data needs to be labeled, and sometimes surveillance video data needs to be labeled, etc.), in order to improve the overall labeling accuracy of the data labeling task, the task publisher of the data labeling task (or the administrator of the crowdsourcing platform) may limit the data labeling task to labelers with relevant type of data labeling experience or labeling capabilities. At this time, the target task executor can represent the above-mentioned labelers with relevant type of data labeling experience or labeling capabilities and their corresponding terminal devices.

[0043] It should be noted that, in the crowdsourcing platform, the data labeling task (that is, the above-mentioned task to be labeled) is a collection of a group of topics to be labeled (that is, equivalent to data to be labeled) of the same type (equivalent to the same field), that is, the above-mentioned task to be labeled is composed of multiple topics to be labeled. The embodiment of the present application does not impose any limitation on the specific number of topics to be labeled contained in a task to be labeled.

[0044] In an embodiment of the present application, as an optional embodiment, the allocation system can obtain the historical data labeling task execution data of the target task executor from the database stored in the crowdsourcing platform in response to the task allocation request for the task to be labeled, and can calculate the data labeling accuracy of the target task executor as of the current moment based on the total number of labeled questions and the number of correctly labeled questions in the historical data labeling task execution data of the target task executor (that is, the data labeling accuracy can represent the proportion of the number of correctly labeled questions by the target task executor in the total number of labeled questions).

[0045] Specifically, when determining the data labeling accuracy of the target task executor according to the above method, if the target task executor can represent any labeler who can perform the data labeling task and his or her corresponding terminal device (that is, the crowdsourcing platform does not limit the labeler's task acceptance authority), then the data labeling accuracy of the target task executor in the general data labeling field can be determined based on the proportion of all questions correctly labeled by the target task executor among all the labeled questions.

[0046] Specifically, if the task publisher of the data labeling task (or the administrator of the crowdsourcing platform) limits the data labeling task to labelers with relevant type of data labeling experience or labeling capabilities, then the target historical data of the same type can be screened out from the labeled historical data of the target executor based on the type of the current task to be assigned (which is also equivalent to the field to which it belongs). Then, based on the proportion of the target historical data correctly labeled by the target task executor in all the labeled target historical data, the data labeling accuracy of the target task executor in the subdivided data labeling field corresponding to the current task to be assigned can be determined.

[0047] Here, after determining the data labeling accuracy of the target task executor, the allocation system can determine the number of assignable questions that matches the above data labeling accuracy as the number of questions that need to be assigned to the target task executor for labeling this time; wherein, the higher the data labeling accuracy of the target task executor, the more the above assignable questions assigned to the target executor, and the lower the data labeling accuracy of the target task executor, the fewer the above assignable questions assigned to the target executor (that is, the correlation direction between the above data labeling accuracy and the above assignable number of questions is positively correlated), so as to enable annotators with high accuracy (that is, high data labeling accuracy) (that is, the target task executor) to participate more in the data labeling task and provide high-quality labeling answers from the number of assigned questions to be labeled, and for annotators with low accuracy (that is, low data labeling accuracy), the low-quality labeling answers provided by low-accuracy annotators can be effectively reduced to avoid low-quality labeling answers interfering with question convergence (that is, causing the questions to be labeled to require more labeling to obtain a convergence result in which the answer distribution obviously tends to one answer).

[0048] S102: If the data annotation accuracy is less than or equal to a preset accuracy threshold, obtain the first number of assignable topics from the multiple topics to be labeled constituting the task to be labeled and assign them to the target task executor.

[0049] It should be noted that the task to be labeled is the data labeling task currently to be assigned, and the question to be labeled is the data to be labeled that needs to be labeled in the task to be labeled; wherein, the question to be labeled can be a multiple-choice question (equivalent to requiring the labeler to select the specific label for this labeling from a limited number of labels), or it can be a numerical question (equivalent to having no options, and the labeler needs to determine the specific label for this labeling by himself); the embodiment of the present application does not impose any limitation on the specific question type to which the question to be labeled belongs.

[0050] Here, considering that the annotation difficulty corresponding to different topics to be labeled under the same annotation task may be different (for example, when the similarity of expression images is also labeled, the image annotation difficulty of selecting an expression image with the greatest difference compared with the other two from three smiling expression images is obviously higher than the image annotation difficulty of selecting an expression image with the greatest difference compared with the other two from two smiling expression images and one crying expression image), the topic to be labeled with a lower annotation difficulty (i.e., the annotation difficulty is less than or equal to a certain annotation difficulty threshold) among the above multiple topics to be labeled can be determined as the first topic to be labeled, and the topic to be labeled with a higher annotation difficulty (i.e., the annotation difficulty is greater than a certain annotation difficulty threshold) among the above multiple topics to be labeled can be determined as the second topic to be labeled.

[0051] Specifically, when the data annotation accuracy of the target task executor is less than or equal to the preset accuracy threshold, it can be determined that the target task executor is a low-accuracy annotator. At this time, according to the number of assignable questions determined in the above step S101, the first number of assignable questions to be labeled with a lower annotation difficulty can be obtained from the above-mentioned first number of questions to be labeled with a lower annotation difficulty and assigned to the target task executor (equivalent to assigning a relatively small number of questions to be labeled but with a lower annotation difficulty to the low-accuracy annotators). On the basis of reducing the low-quality annotation answers provided by low-accuracy annotators, it is also beneficial to improve the data annotation accuracy of low-accuracy annotators, thereby reducing the number of annotations required for the convergence of the answer distribution of the questions to be labeled, while improving the overall annotation efficiency of the task to be labeled and effectively reducing the overall annotation cost of the task to be labeled.

[0052] S103, if the data annotation accuracy is greater than a preset accuracy threshold, obtaining a second number of topics to be labeled that are assignable from a plurality of topics to be labeled constituting the task to be labeled and assigning them to the target task executor.

[0053] Here, referring to the relevant description of the first question to be marked and the second question to be marked in the above step S102, it can be seen that in the embodiment of the present application, the marking difficulty of the second question to be marked is greater than that of the first question to be marked, that is, the second question to be marked is a question to be marked with a higher marking difficulty among the above multiple questions to be marked.

[0054] Specifically, when the data annotation accuracy of the target task executor is greater than a preset accuracy threshold, it can be determined that the target task executor is a high-accuracy annotator. At this time, according to the number of assignable questions determined in the above step S101, the second questions to be labeled with the higher annotation difficulty can be obtained from the second questions to be labeled with the higher annotation difficulty and assigned to the target task executor (equivalent to assigning a relatively large number of questions to be labeled with generally higher annotation difficulty to the annotators with high accuracy), which is conducive to increasing the high-quality annotated answers provided by the high-accuracy annotators and reducing the number of annotations required for the convergence of the answer distribution of the questions to be labeled, thereby improving the overall annotation efficiency of the task to be labeled while effectively reducing the overall annotation cost of the task to be labeled.

[0055] It should be noted that the specific value of the above-mentioned preset accuracy threshold (equivalent to the critical value for determining whether the data annotation accuracy of the target task executor is of high accuracy or low accuracy) can be set according to the actual data annotation task allocation requirements, and the embodiment of the present application does not impose any limitation on this.

[0056] It should be noted that, based on different evaluation dimensions, the evaluation criteria for the difficulty of marking the questions to be marked may be different, that is, the specific measurement criteria for the difficulty of marking the questions to be marked are not unique, and the embodiments of the present application do not make any limitation on this.

[0057] The specific implementation process of each of the above steps in the embodiment of the present application is described in detail below:

[0058] Regarding the specific implementation process of the above-mentioned step S101, in an optional implementation, the task publisher of the task to be labeled (or the administrator of the crowdsourcing platform) can pre-set the number of assignable questions for multiple levels, and the number of assignable questions for each level corresponds to a value range of data labeling accuracy, wherein the higher the data labeling accuracy, the greater the number of assignable questions. At this time, when executing step S101, the number of assignable questions under the target level corresponding to the target value range can be determined according to the target value range of the data labeling accuracy of the target task executor as the number of assignable questions that matches the data labeling accuracy of the target task executor.

[0059] For example, taking the data annotation accuracy between 0 and 30%, the number of assignable topics that match is 30, the data annotation accuracy between 31% and 60%, the number of assignable topics that match is 90, the data annotation accuracy between 61% and 90%, and the number of assignable topics that match is 100, if the data annotation accuracy of the target task executor is 70%, then based on the data annotation accuracy of 70% of the target task executor, it can be determined that the number of assignable topics that match the data annotation accuracy of 70% is 90.

[0060] Regarding the specific implementation process of the above step S101, in another optional implementation, Figure 2 A flow chart of a method for determining the number of allocable questions that match the data annotation accuracy provided by an embodiment of the present application is shown, such as Figure 2 As shown, when executing step S101, the method includes, specifically:

[0061] S201, determining a data labeling accuracy ranking of the target task executor among a plurality of task executors associated with the task to be labeled according to the data labeling accuracy of the target task executor.

[0062] Here, the above-mentioned task executor has executed historical tasks to be labeled, and the historical tasks to be labeled can represent labeling tasks of the same type (that is, the field) as the above-mentioned tasks to be labeled. That is, referring to the relevant instructions about the target task executor in the above-mentioned step S101, in order to improve the quality of data labeling, the task publisher of the data labeling task (or the administrator of the crowdsourcing platform) may limit the data labeling task to labelers with relevant type of data labeling experience or labeling capabilities. At this time, for the current task to be assigned, labelers with the same type of data labeling experience as the task to be labeled can be determined from the crowdsourcing platform as the task executor associated with the task to be labeled (including the target task executor), and according to the data labeling accuracy corresponding to each task executor, the data labeling accuracy ranking of the target task executor among all task executors can be determined.

[0063] As an exemplary explanation, taking the task of labeling image data (e.g., labeling the entity type of entities appearing in the image, each labeling title corresponds to a frame of image to be labeled, etc.) as an example, the allocation system can determine n labelers who have performed image data labeling from the crowdsourcing platform as task executors associated with the task to be labeled (including the target task executor), and then sort the n labelers in descending order according to the data labeling accuracy of the n labelers for labeling the image data (e.g., the data labeling accuracy can be the proportion of correctly labeled historical image data labeling titles in all labeled historical image data labeling titles), and determine that the data labeling accuracy ranking of the target task executor among the n labelers is r% (e.g., if the value of r% is 10%, it means that the data labeling accuracy ranking of the target task executor among the n labelers is in the top 10%).

[0064] It should be noted that, referring to the relevant instructions about the target task executor in the above step S101, if the crowdsourcing platform does not limit the task acceptance authority of the labeler (equivalent to that the labeler can freely accept any type of data labeling task regardless of whether he has experience in labeling data labeling tasks), then the above-mentioned task executors associated with the task to be labeled can be all labelers who have performed data labeling tasks in the crowdsourcing platform; that is, in the embodiment of the present application, the task executors associated with the task to be labeled can essentially represent the labelers and their terminal devices who can perform the task to be labeled (equivalent to the labelers and their terminal devices who can be assigned the topic to be labeled), but because the task publisher of the task to be labeled (or the administrator of the crowdsourcing platform) sets different restrictions on the ability to perform the task to be labeled, the specific task executors associated with the task to be labeled may be different.

[0065] S202, determining a target question quantity level that matches the data annotation accuracy ranking from a plurality of question quantity levels that can be allocated at a single time.

[0066] Here, the multiple question quantity levels can be determined based on the maximum number of questions that can be assigned at one time configured for the task to be labeled (equivalent to the maximum number of questions to be labeled that can be assigned to one labeler at one time) and the preset number of division levels; for example, if the task publisher of the task to be labeled (or the administrator of the crowdsourcing platform) pre-configures the maximum number of questions that can be assigned at one time to be 100, and the preset number of division levels is 5, then it can be determined that the multiple question quantity levels that can be assigned at one time include: the first level (the number of questions to be labeled that can be assigned at one time is 100), the second level (the number of questions to be labeled that can be assigned at one time is 80), the third level (the number of questions to be labeled that can be assigned at one time is 60), the fourth level (the number of questions to be labeled that can be assigned at one time is 40), and the fifth level (the number of questions to be labeled that can be assigned at one time is 20).

[0067] Specifically, as an optional embodiment, the number of preset division levels is recorded as K levels, and the above-mentioned data labeling accuracy ranking corresponding to the target task executor is recorded as r% (that is, the data labeling accuracy ranking of the target task executor among the multiple task executors associated with the task to be labeled is in the top r%), then it can be determined that the target question number level that matches the data labeling accuracy ranking r% is k; wherein, the target question number level k = r% × K, and when the calculation result of k is not an integer, the calculation result of k can be rounded up to obtain an integer calculation result of k (equivalent to k representing the specific target question number level, and the specific value of the level is an integer, so the calculation result of k needs to be rounded up).

[0068] For example, taking the maximum number of questions that can be assigned at one time as 100 and the preset number of division levels as 5 (equivalent to the value of K being 5), if the data labeling accuracy ranking of the target task executor among the multiple task executors associated with the task to be labeled is in the top 39.9% (equivalent to the value of r% being 39.9%), then it can be determined that the value of the target question number level k is 2 (that is, the result of rounding up after k=39.9%×5), that is, the target question number level is the second level mentioned above (the number of questions to be labeled that can be assigned at one time is 80).

[0069] S203, determining the number of questions corresponding to the target question number level as the number of allocatable questions.

[0070] Here, referring to the specific example content in the above step S202, it can be seen that after determining the target question quantity level that matches the above data labeling accuracy ranking corresponding to the target task executor, the number of questions corresponding to the target question quantity level can be directly determined as the number of questions to be labeled that needs to be allocated to the target task executor this time (i.e. the above-mentioned number of allocable questions).

[0071] Specifically, as an optional embodiment, after determining the target number of questions that matches the data annotation accuracy ranking corresponding to the target task executor, the number of questions to be labeled that need to be allocated to the target task executor (i.e., the number of allocable questions) can also be determined according to the following formula:

[0072]

[0073] Wherein, N represents the number of allocable topics mentioned above;

[0074] K represents the number of the above-mentioned preset gear positions;

[0075] k represents the number of the above-mentioned target questions;

[0076] m represents the maximum number of questions that can be assigned at one time.

[0077] For example, taking the maximum number of questions that can be assigned at a single time as 100 (i.e., the value of m is 100) and the number of preset division levels as 5 (equivalent to the value of K being 5), if it is determined that the target question number level that matches the data annotation accuracy ranking corresponding to the target task executor is the second level (equivalent to the value of k being 2), then according to the above formula, it can be calculated that the number of questions to be labeled that need to be assigned to the target task executor this time is 80 (i.e., the calculation result of N is 80, and the calculated number of assignable questions is 80).

[0078] Regarding the specific implementation process of the above steps S102-S103, considering that there is not only one way to quantitatively represent the difficulty of marking a question to be marked by using a specific numerical indicator, based on this, the embodiment of the present application provides at least the following two optional implementation methods for quantitatively representing the difficulty of marking a question to be marked, specifically:

[0079] For each question to be labeled in the task to be labeled, no matter whether the question to be labeled is a multiple-choice question or a numerical question, the labeling result (i.e., the labeled answer) for the question to be labeled can be collected from the annotator who has been assigned the question to be labeled in the crowdsourcing platform. If the answer distribution of the question to be labeled obviously tends to one answer, it can be determined that the question to be labeled has reached convergence (equivalent to taking the above-mentioned answer that obviously tends to be used as the final labeling result of the question to be labeled), and the allocation of the question to be labeled is ended (equivalent to no longer allocating the question to be labeled to the next annotator).

[0080] Based on this, in an optional implementation, for each question to be labeled, multiple existing labeled answers for the question to be labeled (equivalent to the labeled answers fed back by the annotators assigned to the question to be labeled at a historical moment) can be recovered from the crowdsourcing platform, and based on the multiple existing labeled answers for the question to be labeled, truth inference can be performed on the question to be labeled to obtain a truth inference result of the question to be labeled, and the probability that the above truth inference result is in a given interval is the question confidence corresponding to the question to be labeled; wherein, the higher the question confidence, the higher the credibility of the labeling result that the annotator labeled correctly, and the lower the question confidence, the lower the credibility of the labeling result that the annotator labeled correctly; that is, the higher the question confidence, the lower the labeling difficulty corresponding to the question to be labeled, and the lower the question confidence, the higher the standard difficulty corresponding to the question to be labeled.

[0081] Specifically, in the above optional implementation manner, when the marking difficulty of the question to be marked is determined based on the question confidence of the question to be marked, the first question to be marked in the above step S102 (i.e., the question to be marked with a lower marking difficulty) can represent the question to be marked whose question confidence is greater than a preset confidence threshold; and the second question to be marked in the above step S103 (i.e., the question to be marked with a higher marking difficulty) can represent the question to be marked whose question confidence is less than or equal to the above preset confidence threshold.

[0082] It should be noted that the specific value of the above-mentioned preset confidence threshold can be set according to the actual data labeling task allocation requirements, and the embodiments of the present application do not impose any limitation on this.

[0083] In another optional implementation, considering that whether the question to be labeled needs to continue to be assigned depends on whether the answer distribution of the existing annotated answers to the question to be labeled has reached convergence (that is, whether it obviously tends to one answer), therefore, for each question to be labeled, after recovering multiple existing annotated answers to the question to be labeled from the crowdsourcing platform (equivalent to the annotated answers fed back by the annotators who were assigned to the question to be labeled at a historical moment), the uncertainty of the distribution of the answer to the question to be labeled (also equivalent to the degree of dispersion of the existing annotated answers) can also be determined based on the multiple existing annotated answers to the question to be labeled; wherein, a higher degree of uncertainty in the distribution of the answer to the question can indicate a higher difficulty in labeling the question to be labeled, and a lower degree of uncertainty in the distribution of the answer to the question can indicate a lower difficulty in labeling the question to be labeled.

[0084] Specifically, in the above optional implementation manner, when the marking difficulty of the question to be marked is determined according to the uncertainty degree of the answer distribution of the question to be marked, the first question to be marked in the above step S102 (i.e., the question to be marked with a lower marking difficulty) can represent the question to be marked whose answer distribution has a degree of uncertainty less than or equal to the target preset threshold; and the second question to be marked in the above step S103 (i.e., the question to be marked with a higher marking difficulty) can represent the question to be marked whose answer distribution has a degree of uncertainty greater than the above target preset threshold.

[0085] It should be noted that the specific value of the above-mentioned target preset threshold can be set according to the actual data labeling task allocation requirements, and the embodiments of the present application do not impose any limitation on this.

[0086] Here, when determining the difficulty of marking the questions to be marked according to the uncertainty of the distribution of the answers to the questions to be marked, different numerical indicators can be used to quantitatively represent the uncertainty of the distribution of the answers to the questions of different types, specifically:

[0087] When the question to be marked is a multiple-choice question, as an optional embodiment, taking the question to be marked q having X options as an example (that is, the question to be marked has X optional marking options), the entropy value E(q) calculated according to the following entropy value calculation formula can be used to quantitatively represent the uncertainty of the answer distribution of the question to be marked q, specifically:

[0088]

[0089] Where X represents the number of options for the question q to be marked;

[0090] p irepresents the proportion of annotators who selected the i-th option in the existing annotated results of the unannotated question q. For example, if there are 10 annotators who have annotated the unannotated question q, and 3 of them selected the i-th option, then it can be determined that p i The specific value of is 0.3;

[0091] i is used as a count, and its value range is from 1 to X;

[0092] E(q) can represent the entropy value of the distribution of answers to question q to be labeled. The larger the calculated result of the entropy value E(q), the greater the uncertainty of the distribution of answers to question q to be labeled.

[0093] Regarding the above calculation process of the entropy value E(q), it should be noted that, in extreme cases, if the question to be marked is a question assigned for the first time (equivalent to the question to be marked has not been marked by the annotator, and the existing annotated answers to the question to be marked have not been collected), when the question to be marked is a multiple-choice question, the above entropy value E(q) can be calculated in a way that each option has an equal probability of being selected by the annotator (for example, taking the question to be marked as having 10 options, it can be determined that the probability of each option being selected by the annotator is 0.1).

[0094] When the question to be labeled is a multiple-choice question, as an optional embodiment, taking the question q to be labeled as having X options (that is, the question to be labeled has X optional labeling options), the maximum possible information gain G(q) of the answer distribution of the question to be labeled q can also be calculated according to the following formula, and the calculated maximum possible information gain G(q) is used to quantitatively represent the uncertainty of the answer distribution of the question q to be labeled, specifically:

[0095] Under the current answer distribution of the question q to be labeled (i.e., the answer distribution that can be calculated based on the existing marking results of the question q to be labeled when the target task executor has not yet marked the question q to be labeled), it can be assumed that the current target task executor selects the i-th option to mark the question q to be labeled. At this time, under this assumption, the updated answer distribution of the question q to be labeled can be obtained (equivalent to adding the marking result of the target task executor selecting the i-th option when counting the answer distribution of the question q to be labeled), and according to the updated answer distribution and the calculation formula of the above entropy value E(q), the entropy value E(q) of the updated answer distribution of the question q to be labeled can be calculated. ′ :

[0096]

[0097] Where X represents the number of options for the question q to be marked;

[0098] pi ′ represents the proportion of annotators who choose the i-th option in the updated answer distribution of the question to be annotated q. For example, if there are 10 annotators who have annotated the question to be annotated q, and 3 of them choose the i-th option, then in the updated answer distribution of the question, since it is assumed that the target task executor also chooses the i-th option, it can be determined that p i ′ The specific value of

[0100] i is used as a count, and its value range is from 1 to X;

[0101] E(q) ′ It can represent the entropy value of the answer distribution of the question to be labeled q after the question is updated.

[0102] Here, based on the above entropy value E(q) ′ As can be seen from the relevant description, taking the question q to be marked as having X options, i takes values ​​from 1 to X respectively, and the entropy values ​​E(q) under X different assumptions can be calculated: ′ , where the maximum possible information gain G(q) can be expressed as:

[0103] G(q)=max[E(q) ′ ]-min[E(q) ′ ];

[0104] Among them, max[E(q) ′ ] represents the entropy value E(q) under the above X different assumptions ′ The maximum value in ;

[0105] min[E(q) ′ ] represents the entropy value E(q) under the above X different assumptions ′ The minimum value in ;

[0106] The larger the calculated result of the maximum possible information gain G(q), the greater the uncertainty of the distribution of answers to question q to be labeled.

[0107] When the question to be marked is a numerical question, after determining the answer distribution of the question to be marked based on the existing marking results of the question to be marked, the variance corresponding to the answer distribution of the question can be directly calculated, and the calculated variance can be used to quantitatively represent the uncertainty of the answer distribution of the question to be marked; wherein, the larger the calculated variance is, the higher the uncertainty of the answer distribution of the question to be marked is.

[0108] It should be noted that, in extreme cases, if the question to be marked is a question assigned for the first time (equivalent to the question to be marked has not been marked by the annotator, and the existing marked answers to the question to be marked have not been collected), when the question to be marked is a numerical question, it can be assumed that the value of the above variance is infinite.

[0109] In addition, it should be noted that for the above extreme situation, considering that when quantitatively expressing the marking difficulty of the questions to be marked (for example, quantitatively expressing the marking difficulty of the questions to be marked by the above uncertainty method), it is necessary to use the existing marking data of the questions to be marked (that is, the above question answer distribution). Therefore, as an optional embodiment, for the task to be marked (including multiple questions to be marked), it can also be handed over to s labelers in advance before the actual allocation, wherein each labeler randomly labels x questions to be marked in the task to be marked each time (that is, in order to improve the marking efficiency, there is no limit on the need to label all the questions to be marked in the task to be marked). In this way, a certain proportion of the labeled data about the task to be marked can be recovered in advance more conveniently and efficiently, thereby avoiding the above extreme situation, which is conducive to improving the prediction accuracy of the marking difficulty of the questions to be marked in subsequent steps.

[0110] With respect to the specific implementation process of the above step S102, when it is determined that the data annotation accuracy of the target task executor is low (i.e., the data annotation accuracy is less than or equal to the preset accuracy threshold), the target task executor with low data annotation accuracy may be assigned a first to-be-annotated topic with low annotation difficulty according to the method shown in the following step a. Specifically:

[0111] Step a: randomly obtain the first to-be-marked topics with the allocable number of topics from a plurality of first to-be-marked topics and assign them to the target task executor.

[0112] Here, the above-mentioned multiple first questions to be marked belong to the above-mentioned multiple questions to be marked, that is, when it is determined that the data marking accuracy of the target task executor is low (that is, the data marking accuracy is less than or equal to the preset accuracy threshold), the first questions to be marked with lower marking difficulty in the tasks to be marked can be randomly selected. The first questions to be marked that need to be assigned to the target task executor this time only need to ensure that the number of first questions to be marked that are finally assigned is the same as the number of assignable questions determined in the above-mentioned step S101.

[0113] Specifically, the specific method for determining the first question to be marked can refer to the aforementioned optional implementation methods for quantitatively representing the marking difficulty of a question to be marked and the relevant description of the above step S102, and the repeated parts will not be repeated here.

[0114] Regarding the specific implementation process of the above step S102, Figure 3 A flow chart of a method for allocating a second to-be-annotated topic to a target task executor provided in an embodiment of the present application is shown. Figure 3 As shown, when executing step S103, the method includes steps S301-S303, specifically:

[0115] S301, sorting the plurality of second to-be-marked questions in descending order of marking difficulty to obtain a sorted sequence of to-be-marked questions.

[0116] Specifically, the specific method for determining the second question to be marked can refer to the aforementioned optional implementation methods for quantitatively representing the marking difficulty of a question to be marked and the relevant description of the above step S103, and the repeated parts will not be repeated here.

[0117] Here, you can refer to the aforementioned two optional implementation methods for quantitatively representing the marking difficulty of a question to be marked. After quantitatively representing the marking difficulty of a second question to be marked through certain numerical indicators (such as the entropy value, variance and other numerical indicators that can represent the degree of uncertainty in the distribution of the answers to the question to be marked), you can sort the above-mentioned multiple second questions to be marked in order of marking difficulty from high to low (such as sorting the above-mentioned multiple second questions to be marked in order of uncertainty of the answers to the questions from high to low, etc.).

[0118] S302: randomly obtain the first number of second questions to be marked from the second questions to be marked that are ranked before the target ranking in the sequence of questions to be marked and assign them to the target task executor.

[0119] Here, the target ranking is determined based on the data labeling accuracy ranking of the target task executor among the multiple task executors associated with the task to be labeled. For example, referring to the relevant instructions of the previous steps, if the data labeling accuracy ranking of the target task executor among the n task executors is r%, then the above target ranking can represent the second question to be labeled whose labeling difficulty ranking is r% in the sequence of questions to be labeled.

[0120] Here, the correlation direction between the above-mentioned first number of questions and the above-mentioned data labeling accuracy ranking is positively correlated; that is, the higher the above-mentioned data labeling accuracy ranking corresponding to the target task executor, the more difficult problems (i.e., the second questions to be labeled whose labeling difficulty ranking in the sequence of questions to be labeled is before the target ranking) assigned to the target task executor (i.e., the more first questions there are), and the lower the above-mentioned data labeling accuracy ranking corresponding to the target task executor, the fewer difficult problems (i.e., the second questions to be labeled whose labeling difficulty ranking in the sequence of questions to be labeled is before the target ranking) assigned to the target task executor (i.e., the fewer first questions there are).

[0121] S303: Randomly obtain a second number of second questions to be marked from the second questions to be marked that are ranked after the target ranking in the sequence of questions to be marked and assign them to the target task executor.

[0122] Here, the correlation direction between the number of the above-mentioned second questions and the above-mentioned data annotation accuracy ranking is negatively correlated, that is, the higher the above-mentioned data annotation accuracy ranking corresponding to the target task executor, the fewer simple questions (i.e., the second questions to be labeled whose annotation difficulty ranking in the sequence of questions to be labeled is located after the target ranking) allocated to the target task executor (i.e., the fewer the number of the above-mentioned second questions), and the lower the above-mentioned data annotation accuracy ranking corresponding to the target task executor, the more simple questions (i.e., the second questions to be labeled whose annotation difficulty ranking in the sequence of questions to be labeled is located after the target ranking) allocated to the target task executor (i.e., the more the above-mentioned second questions) are.

[0123] It should be noted that since both step S302 and step S303 are steps of allocating second questions to be marked to the target task executor, and the total number of second questions to be marked allocated to the target task executor is the number of allocable questions determined in the above step S101, the sum of the number of second questions in the above step S303 and the number of first questions in the above step S302 is the number of allocable questions determined in the above step S101.

[0124] For example, if the target task executor has a data annotation accuracy ranking of 10% among all task executors and the number of assignable topics determined in step S101 is 100, if the task to be labeled contains 10,000 second topics to be labeled, then after sorting the 10,000 second topics to be labeled in descending order of annotation difficulty, (1-10%)×100 second topics to be labeled can be randomly obtained from the second topics to be labeled whose annotation difficulty ranking is in the top 10% and assigned to the target task executor, and 10%×20% of the second topics to be labeled whose annotation difficulty ranking is after 10% can be randomly obtained from the second topics to be labeled whose annotation difficulty ranking is after 10%. 100 second questions to be labeled are allocated to the target task executor. At this time, the number of the first questions is 90, and the number of the second questions is 10. Therefore, on the basis of ensuring that the total number of second questions to be labeled allocated to the target task executor remains unchanged (that is, it is still the number of assignable questions determined in step S101), more second questions to be labeled with higher labeling difficulty can be allocated to the target task executor with higher data labeling accuracy ranking, which is beneficial to reducing the number of labeling times required for the convergence of the answer distribution of the questions to be labeled, thereby improving the overall labeling efficiency of the task to be labeled and effectively reducing the overall labeling cost of the task to be labeled.

[0125] Regarding the specific implementation process of the above steps S101-S103, after determining the number of questions that need to be assigned to the target task executor this time (i.e., the number of assignable questions in the above step S101), if the determined number of assignable questions is less than the current number of assignable questions to be marked (i.e., the number of questions to be marked in the task to be marked), then the specific questions to be marked that need to be assigned to the target task executor this time can be determined in accordance with the above steps S102-S103.

[0126] In addition, in extreme cases, if there are fewer questions to be marked in the task to be marked, the determined number of allocable questions may be greater than or equal to the current number of allocable questions to be marked (i.e. the number of questions to be marked in the task to be marked). At this time, if the number of allocable questions is greater than or equal to the number of questions to be marked (i.e. the number of questions to be marked in the task to be marked), the allocation system can allocate the task to be marked to the target task executor, that is, allocate all questions to be marked in the task to be marked to the target task executor.

[0127] Regarding the specific implementation process of the above steps S101-S103, considering that whether the to-be-annotated question needs to be continuously assigned to the next annotator depends on whether the distribution of the answers to the to-be-annotated question has reached convergence, and the marking difficulty of the to-be-annotated question is also related to the distribution of the answers to the question (e.g., the higher the uncertainty of the distribution of the answers to the question, the higher the marking difficulty of the to-be-annotated question), in an optional implementation, Figure 4 A flow chart of a method for timely updating the difficulty of marking questions to be marked provided by an embodiment of the present application is shown. Figure 4 As shown, after executing step S103, the method further includes steps S401-S403, specifically:

[0128] S401, judging whether the current answer distribution of the to-be-marked question has reached convergence according to the marked results of the to-be-marked question as of the current moment.

[0129] Here, based on the marked results of the questions to be marked as of the current moment, the answer distribution of the questions to be marked can be determined. If the answer distribution obviously tends to one answer (for example, the proportion of the number of annotations of a certain answer in the total number of annotations exceeds a preset threshold, or exceeds 50%, etc.), it can be determined that the current answer distribution of the questions to be marked has reached convergence; otherwise, it can be determined that the current answer distribution of the questions to be marked has not reached convergence (for example, there is no marked answer whose number of annotations exceeds the preset threshold in the total number of annotations, or there is no marked answer whose number of annotations exceeds 50%, etc.).

[0130] S402: If the current answer distribution of the to-be-marked question reaches convergence, the question allocation for the to-be-marked question is terminated.

[0131] Here, in the data labeling task implemented based on the crowdsourcing platform, when the current answer distribution of a question to be labeled has reached convergence, it can be determined that the labeled answer of the question to be labeled can be determined as the target labeled answer that has reached convergence, that is, it can be determined that the question to be labeled has been labeled and does not need to be assigned to the next labeler for labeling, and the question allocation for the question to be labeled can be ended (for example, the question to be labeled whose answer distribution has reached convergence can be deleted from the questions to be assigned).

[0132] S403: If the current answer distribution of the to-be-marked question has not reached convergence, continue to assign questions to the to-be-marked question, and update the marking difficulty of the to-be-marked question according to the current answer distribution of the to-be-marked question.

[0133] Here, when the current answer distribution of a question to be labeled has not yet reached convergence, it can be determined that the question to be labeled needs to be assigned to the next labeler for labeling until the answer distribution of the question to be labeled reaches convergence.

[0134] In addition, referring to the aforementioned two optional implementation methods for quantitatively representing the marking difficulty of a question to be marked, it can be known that the marking difficulty of the question to be marked can be quantitatively represented according to a certain numerical indicator, and the numerical indicator needs to be calculated based on the latest answer distribution of the question to be marked (refer to the aforementioned specific calculation process of the information gain G(q)). Therefore, based on each allocation of the question to be marked, a marked answer given by the annotator to the question to be marked can be recovered. Therefore, after the question to be marked is allocated, it is necessary to update the answer distribution of the question to be marked according to the latest marked answer recovered, and re-determine the marking difficulty of the question to be marked (that is, update the marking difficulty of the question to be marked) based on the updated answer distribution (that is, the current answer distribution of the question to be marked).

[0135] Based on the above-mentioned method for allocating tasks to be labeled provided in the embodiments of the present application, by improving the way of allocating questions to be labeled, task executors with high data labeling accuracy can be allocated more questions to be labeled with higher labeling difficulty, while task executors with low data labeling accuracy can be allocated relatively fewer questions to be labeled with lower labeling difficulty, which is beneficial to reducing the number of labeling times required for the convergence of the answer distribution of the questions to be labeled, thereby improving the overall labeling efficiency of the tasks to be labeled and effectively reducing the overall labeling cost of the tasks to be labeled.

[0136] Based on the same inventive concept, the present application also provides an allocation device corresponding to the above-mentioned method for allocating tasks to be labeled. Since the principle of solving the problem by the allocation device in the embodiment of the present application is similar to the above-mentioned method for allocating tasks to be labeled in the embodiment of the present application, the implementation of the allocation device can refer to the implementation of the above-mentioned allocation method, and the repeated parts will not be repeated.

[0137] Reference Figure 5 As shown, Figure 5 A schematic diagram of the structure of a device for allocating tasks to be labeled provided in an embodiment of the present application is shown, wherein the device for allocating tasks includes:

[0138] The response module 501 is used to respond to the task assignment request of the task to be labeled, and determine the number of assignable questions that match the data annotation accuracy according to the data annotation accuracy of the target task executor; wherein the correlation direction between the data annotation accuracy and the number of assignable questions is positively correlated;

[0139] A first allocation module 502 is configured to obtain the first number of allocable topics to be labeled from the multiple topics to be labeled constituting the task to be labeled and allocate them to the target task executor if the data labeling accuracy is less than or equal to a preset accuracy threshold;

[0140] The second allocation module 503 is used to obtain the second to-be-annotated questions of the number of assignable questions from the multiple to-be-annotated questions constituting the to-be-annotated task and allocate them to the target task executor if the data annotation accuracy is greater than a preset accuracy threshold; wherein the annotation difficulty of the second to-be-annotated questions is greater than that of the first to-be-annotated questions.

[0141] In an optional implementation, when determining the number of allocatable questions matching the data annotation accuracy according to the data annotation accuracy of the target task executor, the response module 501 is used to:

[0142] According to the data labeling accuracy of the target task executor, determine the data labeling accuracy ranking of the target task executor among multiple task executors associated with the task to be labeled; wherein the task executor has executed historical tasks to be labeled, and the historical tasks to be labeled represent labeling tasks of the same type as the task to be labeled;

[0143] Determine a target question quantity level that matches the data annotation accuracy ranking from a plurality of question quantity levels that can be assigned at a time; wherein the plurality of question quantity levels are determined according to the maximum number of questions that can be assigned at a time configured for the task to be labeled and the number of preset division levels;

[0144] The number of questions corresponding to the target number of questions is determined as the number of allocatable questions.

[0145] In an optional implementation, the first topic to be labeled represents a topic to be labeled whose topic confidence is greater than a preset confidence threshold, and the second topic to be labeled represents a topic to be labeled whose topic confidence is less than or equal to the preset confidence threshold.

[0146] In an optional embodiment, the first question to be labeled represents a question answer distribution whose degree of uncertainty is less than or equal to a target preset threshold of the question to be labeled, and the second question to be labeled represents a question answer distribution whose degree of uncertainty is greater than the target preset threshold of the question to be labeled; wherein, the question answer distribution of the question to be labeled is determined based on the historical labeling results of the question to be labeled.

[0147] In an optional implementation, when obtaining the first to-be-annotated topic of the allocatable number of topics from the plurality of to-be-annotated topics constituting the to-be-annotated task and allocating them to the target task executor, the first allocating module 502 is configured to:

[0148] From a plurality of first to-be-marked topics, first to-be-marked topics of the number of assignable topics are randomly obtained and assigned to the target task executor; wherein the plurality of first to-be-marked topics belong to the plurality of to-be-marked topics.

[0149] In an optional implementation, when obtaining the second to-be-annotated topic with the allocatable number of topics from the multiple to-be-annotated topics constituting the to-be-annotated task and allocating them to the target task executor, the second allocating module 503 is used to:

[0150] Sorting the plurality of second to-be-annotated questions in descending order of difficulty of annotation to obtain an ordered sequence of to-be-annotated questions;

[0151] Randomly obtain the first number of second questions to be labeled from the second questions to be labeled that are ranked before the target ranking in the sequence of questions to be labeled, and assign them to the target task executor; wherein the target ranking is determined according to the data labeling accuracy ranking of the target task executor among the multiple task executors associated with the task to be labeled, and the correlation direction between the first number of questions and the data labeling accuracy ranking is positively correlated;

[0152] Randomly obtain a second number of second questions to be labeled from the second questions to be labeled that are ranked after the target ranking in the sequence of questions to be labeled and assign them to the target task executor; wherein the correlation direction between the second number of questions and the data labeling accuracy ranking is negatively correlated, and the sum of the second number of questions and the first number of questions is the number of assignable questions.

[0153] In an optional embodiment, the distribution device further includes a third distribution module, wherein the third distribution module is used to:

[0154] If the number of assignable topics is greater than or equal to the number of topics to be marked, the tasks to be marked are assigned to the target task executor.

[0155] In an optional embodiment, the allocation device further includes an update module, wherein the update module is used to:

[0156] According to the labeled results of the to-be-labeled question as of the current moment, determining whether the current answer distribution of the to-be-labeled question has reached convergence;

[0157] If the current answer distribution of the to-be-marked question reaches convergence, the question allocation for the to-be-marked question is terminated;

[0158] If the current answer distribution of the to-be-marked question has not reached convergence, continue to assign questions to the to-be-marked question, and update the marking difficulty of the to-be-marked question according to the current answer distribution of the to-be-marked question.

[0159] Based on the above-mentioned device for allocating tasks to be labeled provided in the embodiments of the present application, by improving the method of allocating questions to be labeled, task executors with high data labeling accuracy can be allocated more questions to be labeled with higher labeling difficulty, while task executors with low data labeling accuracy can be allocated relatively fewer questions to be labeled with lower labeling difficulty, which is beneficial to reducing the number of labeling times required for the convergence of the answer distribution of the questions to be labeled, thereby improving the overall labeling efficiency of the tasks to be labeled and effectively reducing the overall labeling cost of the tasks to be labeled.

[0160] Based on the same inventive concept, the present application also provides an electronic device corresponding to the above-mentioned method for allocating tasks to be labeled. Since the principle of solving the problem by the electronic device in the embodiment of the present application is similar to the above-mentioned method for allocating tasks to be labeled in the embodiment of the present application, the implementation of the electronic device can refer to the implementation of the above-mentioned allocation method, and the repeated parts will not be repeated.

[0161] Figure 6A structural diagram of an electronic device 600 provided in an embodiment of the present application includes: a processor 601, a memory 602 and a bus 603, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the electronic device runs a method for allocating tasks to be labeled in an embodiment, the processor 601 communicates with the memory 602 via the bus 603, and the processor 601 executes the machine-readable instructions. When the processor 601 executes the machine-readable instructions, the following steps are implemented, specifically:

[0162] In response to a task assignment request of a task to be labeled, according to the data labeling accuracy of a target task executor, the number of assignable questions matching the data labeling accuracy is determined; wherein the correlation direction between the data labeling accuracy and the number of assignable questions is positively correlated;

[0163] If the data annotation accuracy is less than or equal to a preset accuracy threshold, obtaining the first to-be-annotated topic of the allocatable number of topics from the multiple to-be-annotated topics constituting the to-be-annotated task and assigning them to the target task executor;

[0164] If the data annotation accuracy is greater than a preset accuracy threshold, a second set of topics to be labeled with the number of assignable topics is obtained from a plurality of topics to be labeled constituting the task to be labeled and assigned to the target task executor; wherein the annotation difficulty of the second topic to be labeled is greater than that of the first topic to be labeled.

[0165] In an optional implementation, when determining the number of allocatable questions matching the data annotation accuracy according to the data annotation accuracy of the target task executor, the processor 601 is configured to:

[0166] According to the data labeling accuracy of the target task executor, determine the data labeling accuracy ranking of the target task executor among multiple task executors associated with the task to be labeled; wherein the task executor has executed historical tasks to be labeled, and the historical tasks to be labeled represent labeling tasks of the same type as the task to be labeled;

[0167] Determine a target question quantity level that matches the data annotation accuracy ranking from a plurality of question quantity levels that can be assigned at a time; wherein the plurality of question quantity levels are determined according to the maximum number of questions that can be assigned at a time configured for the task to be labeled and the number of preset division levels;

[0168] The number of questions corresponding to the target number of questions is determined as the number of allocatable questions.

[0169] In an optional implementation, the first topic to be labeled represents a topic to be labeled whose topic confidence is greater than a preset confidence threshold, and the second topic to be labeled represents a topic to be labeled whose topic confidence is less than or equal to the preset confidence threshold.

[0170] In an optional embodiment, the first question to be labeled represents a question answer distribution whose degree of uncertainty is less than or equal to a target preset threshold of the question to be labeled, and the second question to be labeled represents a question answer distribution whose degree of uncertainty is greater than the target preset threshold of the question to be labeled; wherein, the question answer distribution of the question to be labeled is determined based on the historical labeling results of the question to be labeled.

[0171] In an optional implementation, when obtaining the first to-be-annotated topic of the assignable number of topics from the plurality of to-be-annotated topics constituting the to-be-annotated task and assigning it to the target task executor, the processor 601 is configured to:

[0172] From a plurality of first to-be-marked topics, first to-be-marked topics of the number of assignable topics are randomly obtained and assigned to the target task executor; wherein the plurality of first to-be-marked topics belong to the plurality of to-be-marked topics.

[0173] In an optional implementation, when obtaining the second to-be-annotated topic of the assignable number of topics from the plurality of to-be-annotated topics constituting the to-be-annotated task and assigning them to the target task executor, the processor 601 is configured to:

[0174] Sorting the plurality of second to-be-annotated questions in descending order of difficulty of annotation to obtain an ordered sequence of to-be-annotated questions;

[0175] Randomly obtain the first number of second questions to be labeled from the second questions to be labeled that are ranked before the target ranking in the sequence of questions to be labeled, and assign them to the target task executor; wherein the target ranking is determined according to the data labeling accuracy ranking of the target task executor among the multiple task executors associated with the task to be labeled, and the correlation direction between the first number of questions and the data labeling accuracy ranking is positively correlated;

[0176] Randomly obtain a second number of second questions to be labeled from the second questions to be labeled that are ranked after the target ranking in the sequence of questions to be labeled and assign them to the target task executor; wherein the correlation direction between the second number of questions and the data labeling accuracy ranking is negatively correlated, and the sum of the second number of questions and the first number of questions is the number of assignable questions.

[0177] In an optional implementation, after determining the number of assignable questions that matches the data annotation accuracy, the processor 601 is further configured to:

[0178] If the number of assignable topics is greater than or equal to the number of topics to be marked, the tasks to be marked are assigned to the target task executor.

[0179] In an optional implementation, the processor 601 is further configured to:

[0180] According to the labeled results of the to-be-labeled question as of the current moment, determining whether the current answer distribution of the to-be-labeled question has reached convergence;

[0181] If the current answer distribution of the to-be-marked question reaches convergence, the question allocation for the to-be-marked question is terminated;

[0182] If the current answer distribution of the to-be-marked question has not reached convergence, continue to assign questions to the to-be-marked question, and update the marking difficulty of the to-be-marked question according to the current answer distribution of the to-be-marked question.

[0183] Through the above-mentioned electronic device provided in the embodiment of the present application, by improving the allocation method of the questions to be labeled, the task executors with high data labeling accuracy can be allocated more questions to be labeled with higher labeling difficulty, while the task executors with low data labeling accuracy can be allocated relatively fewer questions to be labeled with lower labeling difficulty, which is beneficial to reducing the number of labeling times required for the convergence of the answer distribution of the questions to be labeled, thereby improving the overall labeling efficiency of the tasks to be labeled and effectively reducing the overall labeling cost of the tasks to be labeled.

[0184] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program is executed when a processor is running, and the processor performs the following steps:

[0185] In response to a task assignment request of a task to be labeled, according to the data labeling accuracy of a target task executor, the number of assignable questions matching the data labeling accuracy is determined; wherein the correlation direction between the data labeling accuracy and the number of assignable questions is positively correlated;

[0186] If the data annotation accuracy is less than or equal to a preset accuracy threshold, obtaining the first to-be-annotated topic of the allocatable number of topics from the multiple to-be-annotated topics constituting the to-be-annotated task and assigning them to the target task executor;

[0187] If the data annotation accuracy is greater than a preset accuracy threshold, a second set of topics to be labeled with the number of assignable topics is obtained from a plurality of topics to be labeled constituting the task to be labeled and assigned to the target task executor; wherein the annotation difficulty of the second topic to be labeled is greater than that of the first topic to be labeled.

[0188] In an optional implementation, when determining the number of allocatable questions matching the data annotation accuracy according to the data annotation accuracy of the target task executor, the processor is configured to:

[0189] According to the data labeling accuracy of the target task executor, determine the data labeling accuracy ranking of the target task executor among multiple task executors associated with the task to be labeled; wherein the task executor has executed historical tasks to be labeled, and the historical tasks to be labeled represent labeling tasks of the same type as the task to be labeled;

[0190] Determine a target question quantity level that matches the data annotation accuracy ranking from a plurality of question quantity levels that can be assigned at a time; wherein the plurality of question quantity levels are determined according to the maximum number of questions that can be assigned at a time configured for the task to be labeled and the number of preset division levels;

[0191] The number of questions corresponding to the target number of questions is determined as the number of allocatable questions.

[0192] In an optional implementation, the first topic to be labeled represents a topic to be labeled whose topic confidence is greater than a preset confidence threshold, and the second topic to be labeled represents a topic to be labeled whose topic confidence is less than or equal to the preset confidence threshold.

[0193] In an optional embodiment, the first question to be labeled represents a question answer distribution whose degree of uncertainty is less than or equal to a target preset threshold of the question to be labeled, and the second question to be labeled represents a question answer distribution whose degree of uncertainty is greater than the target preset threshold of the question to be labeled; wherein, the question answer distribution of the question to be labeled is determined based on the historical labeling results of the question to be labeled.

[0194] In an optional implementation, when obtaining the first to-be-annotated topic of the assignable number of topics from the plurality of to-be-annotated topics constituting the to-be-annotated task and assigning it to the target task executor, the processor is configured to:

[0195] From a plurality of first to-be-marked topics, first to-be-marked topics of the number of assignable topics are randomly obtained and assigned to the target task executor; wherein the plurality of first to-be-marked topics belong to the plurality of to-be-marked topics.

[0196] In an optional implementation, when obtaining the second to-be-annotated topic of the number of assignable topics from the plurality of to-be-annotated topics constituting the to-be-annotated task and assigning them to the target task executor, the processor is configured to:

[0197] Sorting the plurality of second to-be-annotated questions in descending order of difficulty of annotation to obtain an ordered sequence of to-be-annotated questions;

[0198] Randomly obtain the first number of second questions to be labeled from the second questions to be labeled that are ranked before the target ranking in the sequence of questions to be labeled, and assign them to the target task executor; wherein the target ranking is determined according to the data labeling accuracy ranking of the target task executor among the multiple task executors associated with the task to be labeled, and the correlation direction between the first number of questions and the data labeling accuracy ranking is positively correlated;

[0199] Randomly obtain a second number of second questions to be labeled from the second questions to be labeled that are ranked after the target ranking in the sequence of questions to be labeled and assign them to the target task executor; wherein the correlation direction between the second number of questions and the data labeling accuracy ranking is negatively correlated, and the sum of the second number of questions and the first number of questions is the number of assignable questions.

[0200] In an optional implementation, after determining the number of assignable questions that matches the data annotation accuracy, the processor is further configured to:

[0201] If the number of assignable topics is greater than or equal to the number of topics to be marked, the tasks to be marked are assigned to the target task executor.

[0202] In an optional implementation, the processor is further configured to:

[0203] According to the labeled results of the to-be-labeled question as of the current moment, determining whether the current answer distribution of the to-be-labeled question has reached convergence;

[0204] If the current answer distribution of the to-be-marked question reaches convergence, the question allocation for the to-be-marked question is terminated;

[0205] If the current answer distribution of the to-be-marked question has not reached convergence, continue to assign questions to the to-be-marked question, and update the marking difficulty of the to-be-marked question according to the current answer distribution of the to-be-marked question.

[0206] Through the above-mentioned computer-readable storage medium provided in the embodiment of the present application, by improving the allocation method of the questions to be labeled, the task executors with high data labeling accuracy can be allocated more questions to be labeled with higher labeling difficulty, while the task executors with low data labeling accuracy can be allocated relatively fewer questions to be labeled with lower labeling difficulty, which is beneficial to reducing the number of labeling times required for the convergence of the answer distribution of the questions to be labeled, thereby improving the overall labeling efficiency of the tasks to be labeled and effectively reducing the overall labeling cost of the tasks to be labeled.

[0207] In an embodiment of the present application, the computer-readable storage medium can also execute other machine-readable instructions when run by the processor to execute the method for allocating tasks to be labeled as described in other embodiments. For the specific steps and principles of the allocation method, please refer to the description of the method side embodiment, which will not be repeated here.

[0208] In the embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0209] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0210] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0211] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0212] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0213] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for allocating tasks to be labeled, characterized in that: The allocation method includes: In response to a task assignment request of a task to be labeled, according to the data labeling accuracy of a target task executor, the number of assignable questions matching the data labeling accuracy is determined; wherein the correlation direction between the data labeling accuracy and the number of assignable questions is positively correlated; If the data annotation accuracy is less than or equal to a preset accuracy threshold, obtaining the first to-be-annotated topic of the allocatable number of topics from the multiple to-be-annotated topics constituting the to-be-annotated task and assigning them to the target task executor; If the data annotation accuracy is greater than a preset accuracy threshold, a second set of topics to be labeled with the number of assignable topics is obtained from a plurality of topics to be labeled constituting the task to be labeled and assigned to the target task executor; wherein the annotation difficulty of the second topic to be labeled is greater than that of the first topic to be labeled.

2. The distribution method according to claim 1, characterized in that: The step of determining the number of allocatable questions that match the data annotation accuracy according to the data annotation accuracy of the target task executor includes: According to the data labeling accuracy of the target task executor, determine the data labeling accuracy ranking of the target task executor among multiple task executors associated with the task to be labeled; wherein the task executor has executed historical tasks to be labeled, and the historical tasks to be labeled represent labeling tasks of the same type as the task to be labeled; Determine a target question quantity level that matches the data annotation accuracy ranking from a plurality of question quantity levels that can be assigned at a time; wherein the plurality of question quantity levels are determined according to the maximum number of questions that can be assigned at a time configured for the task to be labeled and the number of preset division levels; The number of questions corresponding to the target number of questions is determined as the number of allocatable questions.

3. The distribution method according to claim 1, characterized in that: The first topic to be labeled represents a topic to be labeled whose topic confidence is greater than a preset confidence threshold, and the second topic to be labeled represents a topic to be labeled whose topic confidence is less than or equal to the preset confidence threshold.

4. The distribution method according to claim 1, characterized in that: The first question to be labeled represents a question to be labeled whose answer distribution has a degree of uncertainty less than or equal to a target preset threshold, and the second question to be labeled represents a question to be labeled whose answer distribution has a degree of uncertainty greater than the target preset threshold; wherein the answer distribution of the question to be labeled is determined based on the historical labeling results of the question to be labeled.

5. The distribution method according to claim 1, characterized in that: The step of obtaining the first to-be-annotated topic of the allocatable number of topics from the multiple to-be-annotated topics constituting the to-be-annotated task and allocating the first to-be-annotated topic to the target task executor comprises: From a plurality of first to-be-marked topics, first to-be-marked topics of the number of assignable topics are randomly obtained and assigned to the target task executor; wherein the plurality of first to-be-marked topics belong to the plurality of to-be-marked topics.

6. The distribution method according to claim 1, characterized in that: The step of obtaining the second to-be-annotated topics of the number of assignable topics from the multiple to-be-annotated topics constituting the to-be-annotated task and assigning them to the target task executor comprises: Sorting the plurality of second to-be-annotated questions in descending order of difficulty of annotation to obtain an ordered sequence of to-be-annotated questions; Randomly obtain the first number of second questions to be labeled from the second questions to be labeled that are ranked before the target ranking in the sequence of questions to be labeled, and assign them to the target task executor; wherein the target ranking is determined according to the data labeling accuracy ranking of the target task executor among the multiple task executors associated with the task to be labeled, and the correlation direction between the first number of questions and the data labeling accuracy ranking is positively correlated; Randomly obtain a second number of second questions to be labeled from the second questions to be labeled that are ranked after the target ranking in the sequence of questions to be labeled and assign them to the target task executor; wherein the correlation direction between the second number of questions and the data labeling accuracy ranking is negatively correlated, and the sum of the second number of questions and the first number of questions is the number of assignable questions.

7. The distribution method according to claim 1, characterized in that: After determining the number of assignable topics that matches the data annotation accuracy, the assigning method further includes: If the number of assignable topics is greater than or equal to the number of topics to be marked, the tasks to be marked are assigned to the target task executor.

8. The distribution method according to claim 1, characterized in that: The allocation method further comprises: According to the labeled results of the to-be-labeled question as of the current moment, determining whether the current answer distribution of the to-be-labeled question has reached convergence; If the current answer distribution of the to-be-marked question reaches convergence, the question allocation for the to-be-marked question is terminated; If the current answer distribution of the to-be-marked questions has not reached convergence, continue to assign questions to the to-be-marked questions, and update the marking difficulty of the to-be-marked questions according to the current answer distribution of the to-be-marked questions.

9. A device for allocating tasks to be labeled, characterized in that: The dispensing device comprises: A response module, for responding to a task assignment request of a task to be labeled, and determining, according to the data labeling accuracy of a target task executor, the number of assignable questions that matches the data labeling accuracy; wherein the correlation direction between the data labeling accuracy and the number of assignable questions is positively correlated; A first allocation module is configured to obtain the first number of allocable topics to be labeled from the multiple topics to be labeled constituting the task to be labeled and allocate them to the target task executor if the data labeling accuracy is less than or equal to a preset accuracy threshold; A second allocation module is used to obtain second questions to be labeled with the number of allocable questions from multiple questions to be labeled that constitute the task to be labeled and allocate them to the target task executor if the data labeling accuracy is greater than a preset accuracy threshold; wherein the labeling difficulty of the second questions to be labeled is greater than that of the first questions to be labeled.

10. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the method for allocating tasks to be labeled as described in any one of claims 1 to 8 are performed.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for allocating tasks to be labeled as claimed in any one of claims 1 to 8 are executed.

Images