Task return allocation method and device, electronic equipment and storage medium
By calculating the labeling related information of the sub-task, the reasonable allocation of the return of the data labeling task is achieved, the problem of unreasonable returns in the existing technology is solved, and the enthusiasm of the labeler and the development of the data crowdsourcing platform are improved.
Patent Information
- Application Number
- CN202310742659.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-21
- Publication Date
- 2024-12-24
AI Technical Summary
In the prior art, the reward allocation of data annotation tasks is unreasonable, resulting in a decrease in the enthusiasm of the annotator and an increase in the cost of the data crowdsourcing platform.
By calculating the label-related information of the sub-task, including smooth error rate, aggregation ability indicators of the correct labeler and the number of labelers, the weight information of the sub-task is determined, and the task returns are allocated based on the weight information, so as to achieve reasonable allocation based on the task difficulty coefficient.
It has improved the rationality of task return allocation, enhanced the enthusiasm of markers, and promoted the sustainable development of data crowdsourcing platforms.
Smart Images
Figure CN119180429A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a task reward allocation method, device, electronic device and storage medium. Background Art
[0002] With the continuous development of artificial intelligence, data labeling has become an important part of its development process. The accuracy of data labeling directly affects the results of task processing based on artificial intelligence. The data crowdsourcing platform provides a more economical and efficient solution for data labeling.
[0003] Typically, a data labeling task may consist of multiple subtasks. However, in the prior art, all subtasks in a data labeling task are evenly distributed with task rewards, which to a large extent leads to unreasonable distribution of task rewards, thereby reducing the enthusiasm of labelers. Summary of the invention
[0004] The present application provides a task reward allocation method, device, electronic device and storage medium to facilitate the reasonable allocation of task rewards and improve the enthusiasm of annotators.
[0005] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:
[0006] In a first aspect, an embodiment of the present application provides a task reward allocation method, comprising:
[0007] Determine the labeling related information of each subtask under the target labeling task, wherein the labeling related information includes: a smoothed error rate, an aggregation ability index of correct labelers, and the number of labelers of the subtask;
[0008] Determine the weight information of each subtask according to the labeling related information of each subtask, wherein the weight information is used to represent the difficulty coefficient of the subtask in the target labeling task;
[0009] The task reward of each subtask is determined according to the weight information of each subtask and the task reward of the target labeling task.
[0010] In a second aspect, an embodiment of the present application further provides a task reward allocation device, including: a determination module;
[0011] A determination module is used to determine the labeling related information of each subtask under the target labeling task, wherein the labeling related information includes: a smoothed error rate, an aggregation ability index of correct labelers, and the number of labelers of the subtask;
[0012] A determination module, used to determine weight information of each subtask according to the labeling related information of each subtask, wherein the weight information is used to represent the difficulty coefficient of the subtask in the target labeling task;
[0013] The determination module is used to determine the task reward of each subtask according to the weight information of each subtask and the task reward of the target labeling task.
[0014] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to execute the task reward allocation method provided in the first aspect.
[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the task reward allocation method provided in the first aspect is executed.
[0016] The beneficial effects of this application are:
[0017] The present application provides a task reward allocation method, device, electronic device and storage medium. The method obtains the labeling-related information of each subtask under the labeling task to calculate the weight information of each subtask, thereby allocating the task rewards of the subtasks according to the weight information of the subtasks. Since the weight information represents the labeling difficulty coefficient of the subtask, the task rewards are allocated according to the weight information, which realizes the reasonable allocation of task rewards based on the task difficulty coefficient, and avoids the problem of unreasonable allocation caused by the average distribution of task rewards and the reduced labeling enthusiasm of the labelers. Among them, the weight information of the subtask is calculated according to the smoothed error rate of the subtask, the aggregation ability index of the correct labeler and the number of labelers of the subtask. Since the smoothed error rate, the aggregation ability index of the correct labeler and the number of labelers of the subtask are all positively correlated with the difficulty coefficient of the task, the calculated weight information can be used to realize the task reward allocation based on the task difficulty coefficient, improve the rationality of the task reward allocation, and improve the labeling enthusiasm of the task labelers. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 A flowchart of a task reward allocation method provided in an embodiment of the present application;
[0020] Figure 2A flowchart of another task reward allocation method provided in an embodiment of the present application;
[0021] Figure 3 A flowchart of another task reward allocation method provided in an embodiment of the present application;
[0022] Figure 4 A flowchart of another task reward allocation method provided in an embodiment of the present application;
[0023] Figure 5 A flowchart of another task reward allocation method provided in an embodiment of the present application;
[0024] Figure 6 A flowchart of another task reward allocation method provided in an embodiment of the present application;
[0025] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0026] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.
[0027] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0028] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.
[0029] First, the relevant background of this application is explained:
[0030] Data is one of the important elements of artificial intelligence research and practice. In recent years, artificial intelligence research models have become increasingly complex, and the amount of data required has increased accordingly; at the same time, the cross-domains involved in artificial intelligence are becoming more and more extensive, so the demand for field-related data has also gradually increased. Data labeling is an important preparatory work, but it usually faces the problem of large volume and boring process.
[0031] Data crowdsourcing platforms provide a more economical and efficient solution for data labeling, which is to hire a large number of labelers to perform labeling tasks. Since labelers may not have sufficient corresponding professional knowledge or skills for a specific data labeling task, the conventional practice is to assign multiple labelers to each task for labeling, and then infer the true label from the labeling results through the truth inference algorithm.
[0032] Usually, there may be multiple subtasks in a labeling task package. From the perspective of the labeling platform, due to the different difficulties of the subtasks, the task rewards (such as salary compensation) assigned to different subtasks should also be different. However, in the existing task reward allocation, each subtask in the labeling task package is an average task reward. Such unreasonable reward allocation may reduce the enthusiasm of labelers or lead to negative results such as increased platform costs.
[0033] In view of the existence of the above problems, the task reward allocation method provided by the present invention uses the calculated subtask annotation difficulty coefficient as the subtask reward allocation weight, and calculates the reward ratio of each subtask according to the reward allocation weight, thereby realizing reasonable allocation of task rewards based on the different task annotation difficulties, improving the annotation enthusiasm of annotators, and promoting the sustainable development of the data crowdsourcing platform.
[0034] Figure 1 The flowchart of a task reward allocation method provided in the embodiment of the present application is shown below; the execution subject of the method can be a computer device such as a terminal or a server. Figure 1 As shown, the method may include:
[0035] S101, determining the labeling related information of each subtask under the target labeling task, where the labeling related information includes: a smoothed error rate, an aggregation capability index of correct labelers, and the number of labelers of the subtask.
[0036] Usually, for a labeling task, it can be composed of multiple subtasks, and the difficulty coefficients of each subtask may be different. For example, the target labeling task can be: labeling the semantic similarity of sentences; then, each subtask under the target labeling task can be: labeling the semantic similarity of the two given sentences to be labeled. Since the two sentences to be labeled assigned to each subtask are different, when the semantics of the sentence are more complex, the labeling difficulty will be higher, and when the semantics of the sentence are simpler, the labeling difficulty will be relatively lower. Therefore, different sentences to be labeled have different labeling difficulties, resulting in differences in the labeling difficulty of each subtask.
[0037] By performing true value inference on the labeling results of each subtask, the accurate labeling result of the labeling task can be obtained, so that the accurate labeling result of the labeling task can be used as training data in artificial intelligence analysis.
[0038] In this embodiment, the target labeling task can be any labeling task, and the target labeling task can be published through the data crowdsourcing platform. Specifically, each subtask in the target labeling task can be published. It can be published to each pre-determined labeler or published randomly. After each labeler labels the subtask, he can generate corresponding labeling related information, so that the data crowdsourcing platform can collect the labeling related information of each subtask in the target labeling task.
[0039] Optionally, the labeling related information of the subtask may include but is not limited to the following items: smoothed error rate, aggregation ability index of correct labelers, and the number of labelers of the subtask. Among them, the smoothed error rate is used to characterize the labeling error rate of the subtask. When the smoothed error rate is larger, it means that the labeling difficulty of the subtask is greater. The aggregation ability index of the correct labelers is used to characterize the set of labeling abilities of the correct labelers in the subtask. The larger the aggregation ability index of the correct labelers, the higher the ability level of the labelers of the subtask, and the subtask requires more professional knowledge and is more difficult. When the difficulty of the subtask is greater, the labeling results of the labelers are prone to disagreement, resulting in the platform needing to assign more labelers to label in order to achieve convergence conditions. In other words, the greater the difficulty of the subtask, the more labelers are required for the subtask, and the number of labelers for the task is also positively correlated with the difficulty of the task.
[0040] S102 . Determine weight information of each subtask according to the labeling related information of each subtask. The weight information is used to represent the difficulty coefficient of the subtask in the target labeling task.
[0041] Based on the above analysis, since the above-mentioned annotation-related information of the subtask is positively correlated with the difficulty coefficient of the subtask, the weight information of each subtask can be determined separately according to the annotation-related information of each subtask, and the weight information of the subtask represents the difficulty coefficient of the subtask. The larger the difficulty coefficient, the larger the weight information of the subtask. Therefore, when calculating the task reward of the subtask according to the weight information of the subtask, due to the larger weight, the corresponding calculated task reward is also larger.
[0042] S103: Determine the task reward of each subtask according to the weight information of each subtask and the task reward of the target labeling task.
[0043] Usually, the task reward of the target labeling task is pre-allocated, and the task reward of the target labeling task is allocated to each subtask under it. Then, based on the weight information of each subtask, the task reward ratio of each subtask can be obtained, and then the task reward of each subtask is calculated according to the task reward ratio and the task reward of the target labeling task, and the task reward is allocated based on the difficulty coefficient of the subtask. The greater the difficulty coefficient, the higher the task reward ratio, and the higher the allocated task reward. Thus, the purpose of reasonably allocating task rewards is achieved.
[0044] In summary, the task reward allocation method provided in this embodiment obtains the labeling-related information of each subtask under the labeling task to calculate the weight information of each subtask, thereby allocating the task rewards of the subtasks according to the weight information of the subtasks. Since the weight information represents the labeling difficulty coefficient of the subtask, the task rewards are allocated according to the weight information, which realizes the reasonable allocation of task rewards based on the task difficulty coefficient, and avoids the unreasonable distribution caused by the average distribution of task rewards and the problem of reduced labeling enthusiasm of the labelers. Among them, the weight information of the subtask is calculated according to the smoothed error rate of the subtask, the aggregation ability index of the correct labeler, and the number of labelers of the subtask. Since the smoothed error rate, the aggregation ability index of the correct labeler, and the number of labelers of the subtask are all positively correlated with the difficulty coefficient of the task, the calculated weight information can be used to realize the task reward allocation based on the task difficulty coefficient, improve the rationality of the task reward allocation, and improve the labeling enthusiasm of the task labelers.
[0045] Figure 2 A flowchart of another task reward allocation method provided in an embodiment of the present application; Optionally, in step S101, determining the labeling related information of each subtask under the target labeling task includes:
[0046] S201. According to the actual annotation data of the subtask, obtain the number of annotators of the subtask, annotation truth information, and the annotation results of each annotator.
[0047] Optionally, the subtask here may refer to any subtask under the target labeling task. In this embodiment, the weight information calculation method of any subtask is taken as an example for description.
[0048] The actual annotation data of the subtask can be the relevant annotation data of the annotators collected by the data crowdsourcing platform based on the published target annotation task, such as the annotation results, the number of annotators involved in the annotation, and other information.
[0049] According to the actual annotation data, the number of annotators of the subtask can be obtained, which is the total number of people who actually participated in the annotation of the subtask. The annotation truth information can be obtained by calculating the truth value based on the annotation results of each annotator of the subtask, and the annotation truth information is used to represent the standard annotation result of the subtask, which can be understood as the required accurate annotation result.
[0050] S202: Determine the correct annotators and the number of correct annotators according to the annotation truth information and the annotation results of each annotator.
[0051] Optionally, by comparing the annotation truth information with the annotation results of each annotator of the subtask, the correct annotator can be determined from each annotator, and the correct annotator refers to the annotator who correctly annotates the subtask. Based on the determined correct annotators, the number of correct annotators can be determined.
[0052] S203: Determine the smoothed error rate of the subtask according to the number of annotators of the subtask and the number of correct annotators.
[0053] The smoothed error rate of a subtask can be calculated based on the number of annotators for the subtask and the number of correct annotators in the subtask.
[0054] Usually, assuming that the accuracy rate follows the Beta (α, β) distribution, the smoothed error rate can be calculated using the following formula:
[0055]
[0056] Among them, corrcnt t is the number of correct annotators for the subtask, cnt t is the number of annotations for the subtask; when there is no additional prior knowledge, α=β=1 can be taken.
[0057] Among them, the Beta (α, β) distribution can be regarded as a probability distribution of probability. When the specific probability of something is unknown, it can give the likelihood of all probabilities occurring.
[0058] S204: Determine an aggregated capability index of the correct annotators based on the capability level information of each correct annotator.
[0059] The ability level information of the correct annotator can represent the strength of the annotation ability of the correct annotator. The higher the ability level information, the stronger the annotation ability. The aggregate ability index of the correct annotator can be a comprehensive indicator data obtained by integrating the ability level information of each correct annotator. The larger the aggregate ability index of the correct annotator, the stronger the annotation ability of the correct annotator of the subtask, which indicates that the subtask requires stronger professional ability and the greater the annotation difficulty.
[0060] Figure 3 A flowchart of another task reward allocation method provided in an embodiment of the present application; Optionally, in step S202, determining the correct annotator and the number of the correct annotators according to the annotation truth information and the annotation results of each annotator may include:
[0061] S301, determining the correct annotator according to the annotation truth information of the subtask and the annotation results of each annotator of the subtask; the annotation truth information is used to represent the standard annotation result of the subtask.
[0062] The annotation truth information of the subtask can be compared with the annotation results of each annotator of the subtask. When the annotation result of the annotator is consistent with the annotation truth information or the error is within a preset range, the annotator can be determined as the correct annotator. By comparing one by one, all correct annotators of the subtask can be determined.
[0063] S302: Count the correct annotators to obtain the number of correct annotators.
[0064] By counting the number of people who have correctly labeled, the number of people who have correctly labeled can be obtained.
[0065] Optionally, in step S204, determining the aggregated capability index of the correct annotators according to the capability level information of each correct annotator may include: determining the aggregated capability index of the correct annotators according to the annotation-related parameters of each correct annotator in the historical annotation tasks.
[0066] In some embodiments, the aggregation capability index of the correct annotators can be determined based on the annotation related parameters of each correct annotator in the historical annotation tasks. Since each annotator is not limited to performing only one annotation task, each annotator may participate in multiple annotation tasks over time, and the annotation related parameters of each time can be recorded in the background database, so that the annotation related parameters of the correct annotators in the historical annotation tasks can be obtained.
[0067] Figure 4A flowchart of another task reward allocation method provided in an embodiment of the present application; Optionally, in the above steps, determining the aggregation capability index of the correct annotator according to the annotation-related parameters of each correct annotator in the historical annotation task may include:
[0068] S401: Determine the annotation accuracy rate of each correct annotator in the historical annotation task.
[0069] In one achievable manner, the annotation accuracy rate of the correct annotator in the historical annotation task can be used as the ability level information of the correct annotator.
[0070] Of course, in practical applications, the annotation behavior and annotation performance of the correct annotators in the historical annotation tasks can also be used as information about the ability level of the correct annotators.
[0071] S402: taking the average or median of the annotation accuracy of each correct annotator as an aggregation ability indicator of the correct annotators.
[0072] Optionally, the arithmetic mean, geometric mean or median of the annotation accuracy of each correct annotator may be used as the aggregation ability indicator of the correct annotators. It is assumed that the aggregation ability indicator of the correct annotators is represented by aggquality.
[0073] Optionally, in step S102, weight information of each subtask is determined according to the labeling related information of each subtask, which may include: taking the product of the smoothed error rate of each subtask, the aggregation ability index of the correct labeler and the number of labelers as the weight information of each subtask.
[0074] Then, based on the annotation-related information of the subtask calculated above, the weight information of the subtask can be calculated using the following formula:
[0075] x t =cnt t ×incorr t ×aggquality t
[0076] Among them, x t Represents the weight information of subtask t in the target labeling task; cnt t Incorr represents the number of annotations for a subtask; t Characterizes the smoothed error rate of subtask t; aggquality t A measure that characterizes the aggregate ability of the correct annotators for subtask t.
[0077] Figure 5A flowchart of another task reward allocation method provided in an embodiment of the present application; Optionally, in step S103, determining the task reward of each subtask according to the weight information of each subtask and the task reward of the target labeling task may include:
[0078] S501 , normalize the weight information of each subtask to obtain the task reward ratio of each subtask.
[0079] Considering the weight information x of the subtask calculated above t The distribution is between (0, +∞), and in this embodiment, the weight information needs to be scaled to between (0, 1) to convert the task reward ratio. Then, we can first convert x t Shift to (-∞,+∞), and then use the scaling function to scale to get the task reward ratio of the subtask.
[0080] S502: Determine the task reward of each subtask according to the task reward ratio of each subtask and the task reward of the target marking task.
[0081] The task reward ratio refers to the ratio of the task reward of the subtask to the task reward of the target labeling task. Then, the task reward of the subtask can be determined by multiplying the task reward ratio of the subtask by the task reward of the target labeling task.
[0082] Figure 6 A flowchart of another task reward allocation method provided in an embodiment of the present application; Optionally, in step S501, normalizing the weight information of each subtask to obtain the task reward ratio of each subtask may include:
[0083] S601. Generate scaled weight information of the subtask according to the weight information of the subtask, a preset weight scaling ratio and a translation distance; the preset weight scaling ratio is set according to a preset task reward difference range.
[0084] Optionally, the scaled weight information of the subtask may be generated using a preset scaling function according to the weight information of the subtask, a preset weight scaling ratio, and a translation distance.
[0085] In some embodiments, the weight information of the subtask may be scaled using a Logistic function. Specifically, the scaled weight information of the subtask may be calculated using the following formula:
[0086]
[0087] Among them, y tRepresents the scaled weight information of subtask t; k is the scaling ratio (k>0), the larger the k, the larger the scaling ratio, and the more significant the marginal effect decreases; m is the translation distance.
[0088] In some embodiments, the scaling ratio k can be set according to the actual requirements of the labeling task. The reward difference range of different labeling task requirements may be different. The scaling ratio k can be determined according to the reward difference range of tasks with different difficulty levels. The larger k is, the greater the difference between the maximum reward and the minimum reward is.
[0089] S602: Calculate the task reward ratio of the subtask according to the scaled weight information of the subtask and the scaled weight information of each subtask in the target labeling task.
[0090] Optionally, the scaled weight information y of subtask t is t By calculating the ratio of the scaled weight information of each subtask in the target labeling task, the task reward ratio of subtask t can be calculated.
[0091] The task reward ratio of the subtask can be calculated using the following formula:
[0092]
[0093] in, The representation is to sum the scaled weight information of each subtask in the target labeling task.
[0094] Assuming that the task reward of the target labeling task is a, then the task reward ratio w of subtask t is calculated based on t , the task reward of subtask t can be calculated as: a times w t .
[0095] Optionally, in step S601, before generating the scaled weight information of the subtask based on the weight information of the subtask, a preset weight scaling ratio and the translation distance, it can include: calculating and obtaining the translation distance based on the weight information of each subtask, using the maximum value and the minimum value of the weight information in each subtask.
[0096] In this embodiment, two methods for calculating the translation distance are provided.
[0097] The first method is to calculate based on the maximum and minimum weight information of the subtasks in the target labeling task. The following formula can be used for calculation:
[0098]
[0099] Among them, max(x) is the maximum value of the weight information of the subtask in the target labeling task, and min(x) is the minimum value of the weight information of the subtask in the target labeling task.
[0100] When the weight information of each subtask is known, the maximum and minimum values of the weight information can be directly obtained.
[0101] Alternatively, the translation distance is calculated based on the maximum and minimum number of annotators for each subtask, the maximum and minimum smoothing error rate for each subtask, and the theoretical maximum and minimum aggregation ability index of correct annotators.
[0102] In another feasible method, the maximum value max(cnt) and the minimum value min(cnt) of the number of annotators for each subtask in the target labeling task can be determined first. The number of annotators for each subtask can be different. The maximum value of the number of annotators for the subtask is taken as max(cnt), and the minimum value of the number of annotators for the subtask is taken as min(cnt).
[0103] Similarly, the smoothing error rate of each subtask may be different. The maximum value of the smoothing error rates of the subtasks is taken as the maximum smoothing error rate max(incorr), and the minimum value of the smoothing error rates of the tasks is taken as the minimum smoothing error rate min(i corr).
[0104] The aggregation ability index of the correct annotator takes the theoretical maximum value and the theoretical minimum value. When the ability level information of the correct annotator used to calculate the aggregation ability index of the correct annotator is different, the theoretical maximum value and the theoretical minimum value of the aggregation ability index of the correct annotator can also be different. In this embodiment, the aggregation ability index of the correct annotator is calculated based on the annotation accuracy as the ability level information of the correct annotator, and the theoretical maximum value of the annotation accuracy is usually 1, and the minimum value is 0. Therefore, the theoretical maximum value of the aggregation ability index of the correct annotator max(aggquality) can be 1, and the theoretical minimum value of the aggregation ability index of the correct annotator min(aggquality) can be 0.
[0105] Then, the translation distance m can be calculated using the following formula:
[0106]
[0107] Of course, in practical applications, other methods can be used to calculate the translation distance. The above only lists two possible methods.
[0108] Based on the calculated translation distance, the task reward ratio of the subtask can be calculated using the previous task reward ratio calculation method.
[0109] In summary, the task reward allocation method provided in this embodiment obtains the labeling related information of each subtask under the labeling task to calculate the weight information of each subtask, so as to allocate the task rewards of the subtasks according to the weight information of the subtasks. Since the weight information represents the labeling difficulty coefficient of the subtask, the task rewards are allocated according to the weight information, which realizes the reasonable allocation of task rewards based on the task difficulty coefficient, and avoids the problem of unreasonable distribution caused by the average distribution of task rewards and the reduced labeling enthusiasm of the labelers. Among them, the weight information of the subtask is calculated according to the smoothed error rate of the subtask, the aggregation ability index of the correct labeler and the number of labelers of the subtask. Since the smoothed error rate, the aggregation ability index of the correct labeler and the number of labelers of the subtask are all positively correlated with the difficulty coefficient of the task, the calculated weight information can be used to realize the task reward allocation based on the task difficulty coefficient, improve the rationality of the task reward allocation, and improve the labeling enthusiasm of the task labelers.
[0110] The following is a description of the devices, equipment, storage media, etc. used to execute the task reward allocation method provided in this application. The specific implementation process and technical effects are mentioned above and will not be repeated below.
[0111] The embodiment of the present application also provides a task report allocation device, the functions implemented by the task report allocation device correspond to the steps performed by the above method. The device can be understood as the above-mentioned computer device or server, or the processor of the server, or can also be understood as a component independent of the above-mentioned server or processor that implements the functions of the present application under the control of the server, and the device may include: a determination module;
[0112] A determination module is used to determine the labeling related information of each subtask under the target labeling task. The labeling related information includes: smoothed error rate, aggregation ability index of correct labelers, and number of labelers of subtasks;
[0113] A determination module is used to determine the weight information of each subtask according to the labeling related information of each subtask, and the weight information is used to represent the difficulty coefficient of the subtask in the target labeling task;
[0114] The determination module is used to determine the task reward of each subtask according to the weight information of each subtask and the task reward of the target labeling task.
[0115] Optionally, a determination module is specifically used to obtain the number of annotators of the subtask, annotation truth information, and the annotation results of each annotator based on the actual annotation data of the subtask;
[0116] According to the annotation truth information and the annotation results of each annotator, the correct annotators and the number of correct annotators are determined;
[0117] Determine the smoothed error rate of the subtask based on the number of annotations for the subtask and the number of correct annotations;
[0118] According to the ability level information of each correct annotator, the aggregate ability index of the correct annotators is determined.
[0119] Optionally, the determination module is specifically used to determine the correct annotator according to the annotation truth information of the subtask and the annotation results of each annotator of the subtask; the annotation truth information is used to characterize the standard annotation result of the subtask;
[0120] The correct annotators are counted to obtain the number of correct annotators.
[0121] Optionally, the determination module is specifically configured to determine the aggregation capability index of the correct annotators according to the annotation-related parameters of each correct annotator in the historical annotation task.
[0122] Optionally, a determination module is specifically used to determine the annotation accuracy of each correct annotator in the historical annotation task;
[0123] The average or median of the annotation accuracy of each correct annotator is used as the aggregation ability indicator of the correct annotators.
[0124] Optionally, the determination module is specifically configured to use the product of the smoothed error rate of each subtask, the aggregation capability index of the correct annotators, and the number of annotators as the weight information of each subtask.
[0125] Optionally, a determination module is specifically used to normalize the weight information of each subtask to obtain a task reward ratio of each subtask;
[0126] The task reward of each subtask is determined based on the task reward ratio of each subtask and the task reward of the target marking task.
[0127] Optionally, the determination module is specifically used to generate scaled weight information of the subtask according to the weight information of the subtask, a preset weight scaling ratio and a translation distance; the preset weight scaling ratio is set according to a preset task reward difference range;
[0128] The task reward ratio of the subtask is calculated based on the scaled weight information of the subtask and the scaled weight information of each subtask in the target labeling task.
[0129] Optionally, the device may further include: a computing module;
[0130] A calculation module, used to calculate and obtain the translation distance according to the weight information of each subtask, using the maximum value and the minimum value of the weight information in each subtask; or
[0131] The translation distance is calculated based on the maximum and minimum number of annotations for each subtask, the maximum and minimum smoothing error rate for each subtask, and the theoretical maximum and minimum aggregation ability index of the correct annotators.
[0132] Optionally, the determination module is specifically configured to generate scaled weight information of the subtask using a preset scaling function according to the weight information of the subtask, a preset weight scaling ratio, and a translation distance.
[0133] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital singnal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
[0134] The above modules can be connected or communicated with each other via a wired connection or a wireless connection. The wired connection may include a metal cable, an optical cable, a hybrid cable, etc., or any combination thereof. The wireless connection may include a connection in the form of a LAN, a WAN, Bluetooth, a ZigBee, or NFC, or any combination thereof. Two or more modules may be combined into a single module, and any one module may be divided into two or more units. Those skilled in the art will clearly understand that for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application.
[0135] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application includes: a processor 801, a storage medium 802 and a bus 803, wherein the storage medium 802 stores machine-readable instructions executable by the processor 801. When the electronic device runs a task reward allocation method in the embodiment, the processor 801 communicates with the storage medium 802 via the bus 803, and the processor 801 executes the machine-readable instructions to perform the following steps:
[0136] Determine the labeling related information of each subtask under the target labeling task, including: smoothed error rate, aggregation ability index of correct labelers, and number of labelers of subtasks;
[0137] According to the labeling related information of each subtask, the weight information of each subtask is determined respectively, and the weight information is used to represent the difficulty coefficient of the subtask in the target labeling task;
[0138] The task reward of each subtask is determined according to the weight information of each subtask and the task reward of the target labeling task.
[0139] In a feasible implementation scheme, when the processor 801 determines the labeling related information of each subtask under the target labeling task, it is specifically used to: obtain the number of labelers of the subtask, the labeling truth information and the labeling results of each labeler according to the actual labeling data of the subtask;
[0140] According to the annotation truth information and the annotation results of each annotator, the correct annotators and the number of correct annotators are determined;
[0141] Determine the smoothed error rate of the subtask based on the number of annotations for the subtask and the number of correct annotations;
[0142] According to the ability level information of each correct annotator, the aggregate ability index of the correct annotators is determined.
[0143] In a feasible implementation, when the processor 801 determines the correct annotator and the number of the correct annotators according to the annotation truth information and the annotation results of each annotator, it is specifically used to: determine the correct annotator according to the annotation truth information of the subtask and the annotation results of each annotator of the subtask; the annotation truth information is used to characterize the standard annotation result of the subtask;
[0144] The correct annotators are counted to obtain the number of correct annotators.
[0145] In a feasible implementation, when the processor 801 determines the aggregated capability index of the correct annotators according to the capability level information of each correct annotator, it is specifically configured to:
[0146] In a feasible implementation, when the processor 801 determines the aggregated capability index of the correct annotators according to the capability level information of each correct annotator, it is specifically used to: determine the aggregated capability index of the correct annotators according to the annotation related parameters of each correct annotator in the historical annotation tasks.
[0147] In a feasible implementation, when the processor 801 determines the aggregation capability index of the correct annotators according to the annotation related parameters of each correct annotator in the historical annotation task, it is specifically used to: determine the annotation accuracy rate of each correct annotator in the historical annotation task;
[0148] The average or median of the annotation accuracy of each correct annotator is used as the aggregation ability indicator of the correct annotators.
[0149] In a feasible implementation scheme, when the processor 801 determines the weight information of each subtask based on the labeling-related information of each subtask, it is specifically used to: use the product of the smoothed error rate of each subtask, the aggregation ability index of the correct labelers, and the number of labelers as the weight information of each subtask.
[0150] In a feasible implementation scheme, when the processor 801 determines the task reward of each subtask according to the weight information of each subtask and the task reward of the target labeling task, it is specifically used to: normalize the weight information of each subtask to obtain the task reward ratio of each subtask;
[0151] The task reward of each subtask is determined based on the task reward ratio of each subtask and the task reward of the target marking task.
[0152] In a feasible implementation scheme, when the processor 801 performs normalization processing on the weight information of each subtask to obtain the task reward ratio of each subtask, it is specifically used to: generate the scaled weight information of the subtask according to the weight information of the subtask, the preset weight scaling ratio and the translation distance; the preset weight scaling ratio is set according to the preset task reward difference range;
[0153] The task reward ratio of the subtask is calculated based on the scaled weight information of the subtask and the scaled weight information of each subtask in the target labeling task.
[0154] In a feasible implementation scheme, before the processor 801 generates the scaled weight information of the subtask according to the weight information of the subtask, the preset weight scaling ratio and the translation distance, it is specifically used to: calculate and obtain the translation distance according to the weight information of each subtask, using the maximum value of the weight information and the minimum value of the weight information in each subtask; or
[0155] The translation distance is calculated based on the maximum and minimum number of annotations for each subtask, the maximum and minimum smoothing error rate for each subtask, and the theoretical maximum and minimum aggregation ability index of the correct annotators.
[0156] In a feasible implementation scheme, when the processor 801 generates the scaled weight information of the subtask according to the weight information of the subtask, the preset weight scaling ratio and the translation distance, it is specifically used to: generate the scaled weight information of the subtask according to the weight information of the subtask, the preset weight scaling ratio and the translation distance, using a preset scaling function.
[0157] Through the above method, the processing device can obtain the labeling related information of each subtask under the labeling task to calculate the weight information of each subtask, so as to distribute the task reward of the subtask according to the weight information of the subtask. Since the weight information represents the labeling difficulty coefficient of the subtask, the task reward is distributed according to the weight information, which realizes the reasonable distribution of the task reward based on the task difficulty coefficient, and avoids the problem of unreasonable distribution caused by the average distribution of task rewards and the reduced labeling enthusiasm of the labelers. Among them, the weight information of the subtask is calculated according to the smoothed error rate of the subtask, the aggregation ability index of the correct labeler and the number of labelers of the subtask. Since the smoothed error rate, the aggregation ability index of the correct labeler and the number of labelers of the subtask are all positively correlated with the difficulty coefficient of the task, the calculated weight information can be used to realize the task reward distribution based on the task difficulty coefficient, improve the rationality of the task reward distribution, and improve the labeling enthusiasm of the task labelers.
[0158] Among them, the storage medium 802 stores program code, and when the program code is executed by the processor 801, the processor 801 executes various steps in the task reward allocation method according to various exemplary embodiments of the present application described in the above "Exemplary Method" section of this specification.
[0159] Processor 801 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DigitalSignal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware processor to be executed, or a combination of hardware and software modules in the processor can be executed.
[0160] Storage medium 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. The memory is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The storage medium 802 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.
[0161] Optionally, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor performs the following steps:
[0162] Determine the labeling related information of each subtask under the target labeling task, including: smoothed error rate, aggregation ability index of correct labelers, and number of labelers of subtasks;
[0163] According to the labeling related information of each subtask, the weight information of each subtask is determined respectively, and the weight information is used to represent the difficulty coefficient of the subtask in the target labeling task;
[0164] Determine the task rewards of each subtask based on the weight information of each subtask and the task rewards of the target labeling task
[0165] In a feasible implementation scheme, when the processor 801 determines the labeling related information of each subtask under the target labeling task, it is specifically used to: obtain the number of labelers of the subtask, the labeling truth information and the labeling results of each labeler according to the actual labeling data of the subtask;
[0166] According to the annotation truth information and the annotation results of each annotator, the correct annotators and the number of correct annotators are determined;
[0167] Determine the smoothed error rate of the subtask based on the number of annotations for the subtask and the number of correct annotations;
[0168] According to the ability level information of each correct annotator, the aggregate ability index of the correct annotators is determined.
[0169] In a feasible implementation, when the processor 801 determines the correct annotator and the number of the correct annotators according to the annotation truth information and the annotation results of each annotator, it is specifically used to: determine the correct annotator according to the annotation truth information of the subtask and the annotation results of each annotator of the subtask; the annotation truth information is used to characterize the standard annotation result of the subtask;
[0170] The correct annotators are counted to obtain the number of correct annotators.
[0171] In a feasible implementation, when the processor 801 determines the aggregated capability index of the correct annotators according to the capability level information of each correct annotator, it is specifically configured to:
[0172] In a feasible implementation, when the processor 801 determines the aggregated capability index of the correct annotators according to the capability level information of each correct annotator, it is specifically used to: determine the aggregated capability index of the correct annotators according to the annotation related parameters of each correct annotator in the historical annotation tasks.
[0173] In a feasible implementation, when the processor 801 determines the aggregation capability index of the correct annotators according to the annotation related parameters of each correct annotator in the historical annotation task, it is specifically used to: determine the annotation accuracy rate of each correct annotator in the historical annotation task;
[0174] The average or median of the annotation accuracy of each correct annotator is used as the aggregation ability indicator of the correct annotators.
[0175] In a feasible implementation scheme, when the processor 801 determines the weight information of each subtask based on the labeling-related information of each subtask, it is specifically used to: use the product of the smoothed error rate of each subtask, the aggregation ability index of the correct labelers, and the number of labelers as the weight information of each subtask.
[0176] In a feasible implementation scheme, when the processor 801 determines the task reward of each subtask according to the weight information of each subtask and the task reward of the target labeling task, it is specifically used to: normalize the weight information of each subtask to obtain the task reward ratio of each subtask;
[0177] The task reward of each subtask is determined based on the task reward ratio of each subtask and the task reward of the target marking task.
[0178] In a feasible implementation scheme, when the processor 801 performs normalization processing on the weight information of each subtask to obtain the task reward ratio of each subtask, it is specifically used to: generate the scaled weight information of the subtask according to the weight information of the subtask, the preset weight scaling ratio and the translation distance; the preset weight scaling ratio is set according to the preset task reward difference range;
[0179] The task reward ratio of the subtask is calculated based on the scaled weight information of the subtask and the scaled weight information of each subtask in the target labeling task.
[0180] In a feasible implementation scheme, before the processor 801 generates the scaled weight information of the subtask according to the weight information of the subtask, the preset weight scaling ratio and the translation distance, it is specifically used to: calculate and obtain the translation distance according to the weight information of each subtask, using the maximum value of the weight information and the minimum value of the weight information in each subtask; or
[0181] The translation distance is calculated based on the maximum and minimum number of annotations for each subtask, the maximum and minimum smoothing error rate for each subtask, and the theoretical maximum and minimum aggregation ability index of the correct annotators.
[0182] In a feasible implementation scheme, when the processor 801 generates the scaled weight information of the subtask according to the weight information of the subtask, the preset weight scaling ratio and the translation distance, it is specifically used to: generate the scaled weight information of the subtask according to the weight information of the subtask, the preset weight scaling ratio and the translation distance, using a preset scaling function.
[0183] Through the above method, the processing device can obtain the labeling related information of each subtask under the labeling task to calculate the weight information of each subtask, so as to distribute the task reward of the subtask according to the weight information of the subtask. Since the weight information represents the labeling difficulty coefficient of the subtask, the task reward is distributed according to the weight information, and the reasonable distribution of the task reward based on the task difficulty coefficient is realized, and the problem of unreasonable distribution caused by the average distribution of the task reward and the reduced labeling enthusiasm of the labeler is avoided. Among them, the weight information of the subtask is calculated according to the smoothed error rate of the subtask, the aggregation ability index of the correct labeler, and the number of labelers of the subtask. Since the smoothed error rate, the aggregation ability index of the correct labeler, and the number of labelers of the subtask are all positively correlated with the difficulty coefficient of the task, the calculated weight information can be used to realize the task reward distribution based on the task difficulty coefficient, improve the rationality of the task reward distribution, and improve the labeling enthusiasm of the task labeler. In the embodiment of the present application, the computer program can also execute other machine-readable instructions when it is run by the processor to execute the method described in other embodiments. For the specific execution method steps and principles, please refer to the description of the embodiment, which will not be repeated in detail here.
[0184] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0185] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0186] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0187] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.
Claims
1. A task reward allocation method, characterized in that: include: Determine the labeling related information of each subtask under the target labeling task, wherein the labeling related information includes: a smoothed error rate, an aggregation ability index of correct labelers, and the number of labelers of the subtask; Determine the weight information of each subtask according to the labeling related information of each subtask, wherein the weight information is used to represent the difficulty coefficient of the subtask in the target labeling task; The task reward of each subtask is determined according to the weight information of each subtask and the task reward of the target labeling task.
2. The method according to claim 1, characterized in that: The labeling related information of each subtask under the target labeling task is determined, including: According to the actual annotation data of the subtask, obtain the number of annotators for the subtask, annotation truth information, and the annotation results of each annotator; Determine the correct annotators and the number of correct annotators according to the annotation truth information and the annotation results of each annotator; Determining a smoothed error rate of the subtask according to the number of people who annotated the subtask and the number of people who correctly annotated the subtask; According to the ability level information of each correct annotator, an aggregated ability index of the correct annotators is determined.
3. The method according to claim 2, characterized in that The determining the correct annotators and the number of the correct annotators according to the annotation truth information and the annotation results of each annotator includes: Determine the correct annotator according to the annotation truth information of the subtask and the annotation results of each annotator of the subtask; the annotation truth information is used to characterize the standard annotation result of the subtask; The correct annotators are counted to obtain the number of correct annotators.
4. The method according to claim 2, characterized in that: Determining the aggregated capability index of the correct annotators according to the capability level information of each correct annotator includes: According to the annotation related parameters of each correct annotator in the historical annotation task, the aggregation capability index of the correct annotator is determined.
5. The method according to claim 4, characterized in that Determining the aggregation capability index of each correct annotator according to the annotation related parameters of each correct annotator in the historical annotation task includes: Determine the annotation accuracy of each correct annotator in the historical annotation task; The average value or median of the annotation accuracy of the correct annotators is used as the aggregation ability indicator of the correct annotators.
6. The method according to any one of claims 1 to 5, characterized in that: Determining the weight information of each subtask according to the labeling related information of each subtask includes: The product of the smoothed error rate of each subtask, the aggregation ability index of the correct annotators, and the number of annotators is used as the weight information of each subtask.
7. The method according to claim 6, characterized in that Determining the task reward of each subtask according to the weight information of each subtask and the task reward of the target labeling task includes: Normalizing the weight information of each subtask to obtain the task reward ratio of each subtask; The task rewards of each subtask are determined according to the task reward ratios of each subtask and the task reward of the target labeling task.
8. The method according to claim 7, characterized in that The weight information of each subtask is normalized to obtain the task reward ratio of each subtask, including: Generate scaled weight information of the subtask according to the weight information of the subtask, a preset weight scaling ratio and a translation distance; the preset weight scaling ratio is set according to a preset task reward difference range; The task reward ratio of the subtask is calculated according to the scaled weight information of the subtask and the scaled weight information of each subtask in the target labeling task.
9. The method according to claim 8, characterized in that Before generating the scaled weight information of the subtask according to the weight information of the subtask, the preset weight scaling ratio and the translation distance, the method includes: According to the weight information of each subtask, the maximum value and the minimum value of the weight information in each subtask are used to calculate and obtain the translation distance; or The translation distance is calculated based on the maximum and minimum values of the number of annotators for each subtask, the maximum and minimum values of the smoothing error rate for each subtask, and the theoretical maximum and minimum values of the aggregation ability index of the correct annotators.
10. The method according to claim 8, characterized in that The step of generating scaled weight information of the subtask according to the weight information of the subtask, a preset weight scaling ratio, and a translation distance includes: According to the weight information of the subtask, the preset weight scaling ratio and the translation distance, a preset scaling function is used to generate the scaled weight information of the subtask.
11. A task reward distribution device, characterized in that: include: Identify the module; The determination module is used to determine the labeling related information of each subtask under the target labeling task, and the labeling related information includes: smoothed error rate, aggregation ability index of correct labelers, and number of labelers of the subtask; The determination module is used to determine the weight information of each subtask according to the labeling related information of each subtask, and the weight information is used to represent the difficulty coefficient of the subtask in the target labeling task; The determination module is used to determine the task reward of each subtask according to the weight information of each subtask and the task reward of the target labeling task.
12. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the program instructions to execute the task reward allocation method as described in any one of claims 1 to 10.
13. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the task reward allocation method as described in any one of claims 1 to 10 is executed.