Contribution degree evaluation system and method for online crowdsourcing system

Abstract

According to the technical scheme, the contribution degree evaluation system of the online crowdsourcing system is characterized by comprising a task management and storage module, a task progress evaluation module and a task contribution degree evaluation module which are connected in sequence. On the other hand, the invention provides a contribution degree evaluation method of the online crowdsourcing system. According to the method, submission of subpackage tasks can be stimulated, and the completion speed of crowdsourcing tasks can be accelerated; and because the contribution degree follows distribution of which the sequence is gradually reduced from old to new when being distributed in each type of tasks, the contribution degree of the subpackage tasks submitted firstly is greater than that of the subpackage tasks submitted later in the same type. If the crowdsourcing platform provides rewards for the user according to the contribution degree of submitting the subpackage task, the contribution degree evaluation system can bring more rewards for the crowdsourcing platform user submitting the subpackage task firstly, so that the user is motivated to complete and submit the subpackage task as soon as possible.

Description

Technical Field

[0001] The present invention relates to a contribution evaluation system and method for an online crowdsourcing system, belonging to the field of crowdsourcing rewards and contribution distribution. Background Art

[0002] With the development of Internet technology, crowdsourcing has become a new type of cooperation method on the Internet. It enables Party A with demands to publicly collect temporary labor on the Internet by posting crowdsourcing tasks on the web page or client of the crowdsourcing platform, and participants can submit corresponding sub-tasks on the crowdsourcing platform to help complete the crowdsourcing tasks. How to provide rewards that match the work contributions for crowdsourcing participants has become an increasingly important issue.

[0003] Traditional reward methods often have the following problems: 1. The amount of the reward is fixed, so it cannot match the contributions of crowdsourcing participants; 2. The reward distribution can only be carried out after the crowdsourcing task is completed. If the time period of the crowdsourcing task is long, participants need to wait for a long time to obtain the reward. Summary of the Invention

[0004] The object of the present invention is to: measure the contribution degree of each sub-task submitted by each participant in real time, and provide a basis for reward distribution.

[0005] In order to achieve the above object, the technical solution of the present invention discloses a contribution evaluation system for an online crowdsourcing system, which is characterized in that it includes a task management and storage module, a task progress evaluation module, and a task contribution evaluation module connected in sequence, wherein:

[0006] The task management and storage module is used to obtain the original information of the crowdsourcing task, the information of the already submitted sub-tasks from the sub-task submission terminal of the online crowdsourcing platform and store them, wherein the online crowdsourcing platform splits the crowdsourcing task into several sub-tasks;

[0007] The task progress evaluation module is used to quantitatively calculate the completion progress of the current task;

[0008] The task contribution evaluation module is used to calculate in real time the importance of the sub-tasks submitted by each participant of each crowdsourcing task to the total task to obtain the contribution degree of each sub-task, and the reward distribution terminal of the online crowdsourcing platform provides rewards for each participant of each crowdsourcing task according to the contribution degree.

[0009] Preferably, the original information of the crowdsourcing task includes the task name, the category of the required sub-tasks, and the number of each required sub-task.

[0010] Preferably, the information of the already submitted sub-tasks includes the category of each sub-task collected by the online crowdsourcing platform and the submission time.

[0011] Preferably, the completion progress of the current task is a percentage indicating the degree of task completion, or an amount in currency representing the value of the current task.

[0012] Preferably, the contribution degree is a percentage reflecting the contribution ratio of the subcontracted task, or an amount in currency representing the value of the subcontracted task.

[0013] Preferably, the task contribution degree evaluation module comprehensively evaluates the contribution degree of each current subcontracted task in real time by integrating the original information of the crowdsourcing task, the information of the already submitted subcontracted tasks, and the current task progress, and updates the contribution degree of each subcontracted task when a new subcontracted task is received on the online crowdsourcing platform.

[0014] Preferably, a task classification module is integrated in the task contribution degree evaluation module: if the subcontracted tasks released on the crowdsourcing platform do not have a clear classification, resulting in no category information of the tasks in the information of the already submitted subcontracted tasks, preprocessing can be performed through the task classification module to achieve task classification, and then the contribution degree is determined through the task contribution degree evaluation module.

[0015] Preferably, a subcontracted task review module is embedded in the task management and storage module to set a threshold for the completion quality of each subcontracted task: if it is higher than or equal to the threshold, the current subcontracted task is marked as "completed"; if it is lower than the threshold, the current subcontracted task is marked as "uncompleted";

[0016] The subcontracted tasks marked as uncompleted are ignored by the task progress evaluation module and the task contribution degree evaluation module, and this information is sent back to the crowdsourcing task platform.

[0017] Another aspect of the present invention is to provide a method for evaluating the contribution degree of an online crowdsourcing system, which is characterized in that it is implemented based on the above-mentioned contribution degree evaluation system of the online crowdsourcing system, and includes the following steps:

[0018] Step 1: After the crowdsourcing task is released, the task management and storage module obtains the original information of the crowdsourcing task from the submission terminal of the online crowdsourcing platform; the task management and storage module sends the original information of the crowdsourcing task to the task progress evaluation module;

[0019] Step 2: After each new subcontracted task is submitted, the task management and storage module obtains the submitted subcontracted task information from the submission terminal of the online crowdsourcing platform and stores the subcontracted task information locally; the task management and storage module sends all the stored information of the already submitted subcontracted tasks to the task progress evaluation module;

[0020] Step 3: The task progress evaluation module integrates the original information of the crowdsourcing tasks obtained and the information of the subcontracted tasks that have been submitted, evaluates the completion status of the current task, thereby obtaining the task completion progress, and sends the task completion progress and the information of the subcontracted tasks that have been submitted to the task contribution evaluation module;

[0021] Step 4: The task contribution evaluation module extracts the category information of each task from the information of the subcontracted tasks that have been submitted;

[0022] The task contribution evaluation module divides the subcontracted tasks that have been submitted into several categories;

[0023] The task contribution evaluation module calculates the marginal contribution of each subcontracted task to the completion degree of the crowdsourcing task at the time of submission. Among them, the marginal contribution of the subcontracted task refers to the change in the task completion degree before and after the submission of this subcontracted task;

[0024] The task contribution evaluation module calculates the sum of the marginal contributions of each category of tasks respectively, and assigns contributions to each task in decreasing order from old to new. Among them, when assigning contributions to each task in each category of tasks, the total contribution obtained by this category of tasks should be equal to the sum of the marginal contributions of each task in this category of tasks.

[0025] Preferably, after each new subcontracted task is submitted, the information of the subcontracted task is stored locally through the task management and storage module, and the task management and storage module is connected to the task contribution evaluation module. The task contribution evaluation module directly reads the information of the subcontracted tasks that have been submitted from the task management and storage module.

[0026] The present invention provides a contribution evaluation system for an online crowdsourcing platform, determines the importance of each subcontracted task, and thereby formulates rewards provided to each crowdsourcing participant. The present invention can motivate the submission of subcontracted tasks and accelerate the completion speed of crowdsourcing tasks: Since the contribution is distributed in decreasing order from old to new when assigned in each category of tasks, the contribution of the subcontracted tasks submitted earlier is greater than that of the subcontracted tasks submitted later in the same category. If the crowdsourcing platform provides rewards to users according to the contribution of the submitted subcontracted tasks, then the contribution evaluation system will bring more rewards to the crowdsourcing platform users who submit subcontracted tasks earlier, thereby motivating users to complete and submit subcontracted tasks as soon as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Illustrates an example of applying the present invention;

[0028] Figure 2 Illustrates the connection relationship between the task submission terminal of the crowdsourcing platform and the present invention;

[0029] Figure 3Schematically shows the implementation process of the present invention. Specific embodiments

[0030] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

[0031] As Figure 1 shown, in an embodiment of the present invention, Company A intends to train an artificial intelligence model that can distinguish pictures of cats and dogs, and label all pictures of cats as cats and all pictures of dogs as dogs. To achieve this goal, Company A needs to collect a large number of pictures of cats and dogs that have been correctly labeled as training data, so it chooses to publish a crowdsourcing task on crowdsourcing platform B and pays a certain amount to crowdsourcing platform B. The original information of this crowdsourcing task is: need to collect pictures of cats and dogs until the accuracy rate of the artificial intelligence model reaches a0. After receiving the request, crowdsourcing platform B splits the crowdsourcing task into several sub-tasks and publishes two sub-tasks that users can upload: 1. Upload pictures of cats; 2. Upload pictures of dogs. Users who have registered on crowdsourcing platform B can view the sub-tasks published by crowdsourcing platform B and upload the sub-tasks. After receiving the sub-tasks, crowdsourcing platform B records the records of users uploading sub-tasks and provides rewards to users. In addition, crowdsourcing platform B may also provide additional rewards to users who have submitted sub-tasks in the future.

[0032] As Figure 2 shown, the task submission terminal of crowdsourcing platform B is connected to the task management and storage module of a contribution evaluation system of an online crowdsourcing system disclosed in the present invention. The task management and storage module is used to obtain the original information of the crowdsourcing task and the information of the submitted sub-tasks from the sub-task submission terminal of crowdsourcing platform B. For the example as Figure 1 shown, in conjunction with Figure 3 , in an embodiment of the present invention:

[0033] First, after the crowdsourcing platform B accepts the crowdsourcing task from Company A and publishes it, the task management and storage module of the present invention obtains the original information of the crowdsourcing task from the subcontract task submission terminal of the crowdsourcing platform B, that is, "need to collect pictures of cats and dogs until the accuracy rate of the artificial intelligence model C reaches a0". In order for the task progress evaluation module to calculate the completion progress of the task, in this embodiment, the task management and storage module needs to request an interface from the crowdsourcing platform B that can obtain the current accuracy rate of the artificial intelligence model C, and this interface will be used as a part of the original information of the crowdsourcing task. After each user submits a subcontract task, the task management and storage module obtains the submitted subcontract task information from the subcontract task submission terminal. In the embodiment of the present invention, the subcontract task information includes: the submitting user, the subcontract task category, and the pictures uploaded by the user, etc. The task management and storage module stores the above information locally.

[0034] Further, the task progress evaluation module of the present invention obtains the original information of the crowdsourcing task and the uploaded subcontract task information from the task management and storage module. Here, a1 represents the current accuracy rate of the artificial intelligence model C. In this embodiment, the task progress evaluation module obtains the current accuracy rate a1 from the accuracy rate interface of the artificial intelligence model C, and obtains the target accuracy rate a0 from the original information of the crowdsourcing task. Take a1 / a0 as the completion progress of the current task. When a1 = a0, the task completion progress is 100%, corresponding to the accuracy rate of the artificial intelligence model C reaching a0 desired by Company A.

[0035] Further, the task contribution evaluation module of the present invention obtains the uploaded subcontract task information and the task completion progress from the task progress evaluation module. The task contribution evaluation module is used to calculate the contribution of each subcontract task. The task contribution evaluation module saves the task completion progress locally, so the task contribution evaluation module has the task completion progress after each subcontract task is submitted. The task contribution evaluation module first calculates the marginal contribution of each subcontract task to the task completion progress, that is, the task completion progress a after the submission of this subcontract task ′The difference between 1 / a0 and the task completion progress a1 / a0 before submission. After that, the task contribution evaluation module calculates the total marginal contribution of the sub-tasks of each type of submission. In this embodiment, the task contribution module sums up the marginal contributions of all the sub-tasks of "submitting pictures of cats" as the total marginal contribution of this type of sub-tasks; sums up the marginal contributions of all the sub-tasks of "submitting pictures of dogs" as the total marginal contribution of this type of sub-tasks. Finally, the task contribution evaluation module redistributes the total marginal contribution of each type of sub-tasks within this type of tasks in a decreasing order from the old to the new. In this embodiment, a method of distributing contributions is as follows: The total marginal contribution of the sub-tasks of "submitting pictures of cats" is counted as c. The contribution degree of c / 2 is assigned to the first sub-task of submitting pictures of cats, the contribution degree of c / 4 is assigned to the second sub-task of submitting pictures of cats, and so on until the penultimate sub-task, and the last sub-task is assigned all the remaining contribution degrees. After the task contribution evaluation module completes the contribution evaluation, it sends all the sub-tasks and their contribution degree distributions to the crowdsourcing platform. The crowdsourcing platform provides rewards for each user according to the contribution degree; for some users, their contribution degree may increase after submitting new sub-tasks, and the platform can provide supplementary rewards for this type of users.

[0036] As an improvement of the present invention, a task review module and a task classification module can be respectively integrated into the task management and storage module and the task contribution evaluation module. The task review module is used to confirm whether the sub-tasks submitted by users meet the quality requirements. If the sub-tasks do not meet the quality requirements, they will not be adopted and their contribution degrees will not be calculated. In this embodiment, an implementation method of the task review module is to obtain information on whether the sub-tasks are qualified from Company A: After Company A collects data, it often conducts a review within the company to judge whether the pictures meet the requirements and whether the cat-dog classification marked is correct. For sub-tasks that do not meet the quality requirements, the task management and storage module discards them. The task classification module is used to classify the sub-tasks submitted by users. In this embodiment, the submitted sub-tasks have been classified, so there is no need for the task classification module to work; in other application scenarios of the present invention, it may occur that the sub-tasks are not classified when collected, and the task classification module can classify them first and then evaluate their contributions.

[0037] In this embodiment, the sub-tasks submitted earlier have a higher contribution degree than those submitted later of the same type. Therefore, users are motivated to submit the sub-tasks they have completed as soon as possible. Therefore, the present invention can improve the speed of the crowdsourcing platform to complete crowdsourcing tasks.

Claims

1. A contribution evaluation system for an online crowdsourcing system, characterized in that, It includes a sequentially connected task management and storage module, a task progress evaluation module, and a task contribution evaluation module, where: The task management and storage module is used to obtain the original information of crowd-sourcing tasks, the information of the submitted sub-tasks, and store them from the sub-task submission terminal of the online crowd-sourcing platform. Here, the online crowd-sourcing platform splits the crowd-sourcing tasks into several sub-tasks; The task progress evaluation module is used to quantitatively calculate the completion progress of the current task; The task contribution evaluation module is used to calculate in real time the importance of the sub-tasks submitted by each participant in a crowd-sourcing task to the total task for the online crowd-sourcing platform, so as to obtain the contribution degree of each sub-task. The reward distribution terminal of the online crowd-sourcing platform provides rewards for each participant in the crowd-sourcing task according to the contribution degree.

2. The contribution degree evaluation system of an online crowdsourcing system according to claim 1, characterized in that The original information of the crowd-sourcing task includes the task name, the category of the required sub-tasks, and the number of each required sub-task.

3. The contribution degree evaluation system of an online crowdsourcing system according to claim 1, characterized in that, The information of the submitted sub-tasks includes the category and submission time of each sub-task collected by the online crowd-sourcing platform.

4. The contribution degree evaluation system of an online crowdsourcing system according to claim 1, characterized in that, The completion progress of the current task is a percentage indicating the degree of task completion, or an amount in currency indicating the value of the current task.

5. The contribution degree evaluation system of an online crowdsourcing system according to claim 1, wherein The contribution degree is a percentage reflecting the contribution ratio of the sub-task, or an amount in currency indicating the value of the sub-task.

6. The contribution degree evaluation system of an online crowdsourcing system according to claim 1, wherein The task contribution evaluation module evaluates the contribution degree of each current sub-task in real time by integrating the original information of the crowd-sourcing task, the information of the submitted sub-tasks, and the current task progress, and updates the contribution degree of each sub-task when the online crowd-sourcing platform receives a new sub-task.

7. The contribution degree evaluation system of an online crowdsourcing system according to claim 1, characterized in that, A task classification module is integrated in the task contribution evaluation module: If the sub-tasks released by the crowd-sourcing platform do not have a clear classification, resulting in no category information of the tasks in the information of the submitted sub-tasks, preprocessing can be performed through the task classification module to achieve task classification, and then the contribution degree is determined through the task contribution evaluation module.

8. The contribution degree evaluation system of an online crowdsourcing system according to claim 1, characterized in that, A sub-task review module is embedded in the task management and storage module to set a threshold for the completion quality of each sub-task: If it is higher than or equal to the threshold, the current sub-task is marked as "completed"; If it is lower than the threshold, the current sub-task is marked as "uncompleted"; The sub-tasks marked as uncompleted are ignored by the task progress evaluation module and the task contribution evaluation module, and the information is sent back to the crowd-sourcing task platform.

9. A method for evaluating the contribution degree of an online crowdsourcing system, characterized in that, It is implemented based on the contribution evaluation system of the online crowd-sourcing system described in claim 1, including the following steps: Step 1: After the crowd-sourcing task is released, the task management and storage module obtains the original information of the crowd-sourcing task from the submission terminal of the online crowd-sourcing platform; the task management and storage module sends the original information of the crowd-sourcing task to the task progress evaluation module; Step 2: After each new sub-task is submitted, the task management and storage module obtains the submitted sub-task information from the submission terminal of the online crowd-sourcing platform and stores the sub-task information locally; the task management and storage module sends all the stored information of the submitted sub-tasks to the task progress evaluation module; Step 3: The task progress evaluation module integrates the original information of the crowdsourcing tasks obtained and the information of the subcontracted tasks that have been submitted, evaluates the completion status of the current task, thereby obtaining the task completion progress, and sends the task completion progress and the information of the subcontracted tasks that have been submitted to the task contribution evaluation module; Step 4: The task contribution evaluation module extracts the category information of each task from the information of the subcontracted tasks that have been submitted; The task contribution evaluation module divides the subcontracted tasks that have been submitted into several categories; The task contribution evaluation module calculates the marginal contribution of each subcontracted task to the completion of the crowdsourcing task at the time of submission, where the marginal contribution of the subcontracted task refers to the change in the task completion degree before and after the submission of the subcontracted task; The task contribution evaluation module calculates the sum of the marginal contributions of each category of tasks respectively, and allocates contributions to each task in decreasing order from old to new. When allocating contributions to each task in each category of tasks, the total contribution obtained by this category of tasks should be equal to the sum of the marginal contributions of each task in this category of tasks.

10. The contribution degree evaluation method of an online crowdsourcing system according to claim 9, characterized in that, After each new subcontracted task is submitted, the information of the subcontracted task is stored locally through the task management and storage module, and the task management and storage module is connected to the task contribution evaluation module. The task contribution evaluation module directly reads the information of the subcontracted tasks that have been submitted from the task management and storage module.

Images