Evaluating workers in crowdsourcing environment

Abstract

A crowdsourcing environment is described herein which uses a single-stage or multi-stage approach to evaluate the quality of work performed by a worker, with respect to an identified task. In the multi-stage case, an evaluation system, in the first stage, determines whether the worker corresponds to a spam agent. In a second stage, for a non-spam worker, the evaluation system determines the propensity of the worker to perform desirable (e.g., accurate) work in the future. The evaluation system operates based on a set of features, including worker-focused features (which describe work performed by the particular worker), task-focused features (which describe tasks performed in the crowdsourcing environment), and system-focused features (which describe aspects of the configuration of the crowdsourcing environment). According to one illustrative aspect, the evaluation system performs its analysis using at least one model, produced using any type of supervised machine learning technique.

Description

Background technique [0001] A computer-implemented crowdsourcing system operates by distributing task instances to a population of human workers and then collecting workers' responses to the task. In some cases, the crowdsourcing system may reward the worker for his or her individual contributions on behalf of the entity sponsoring or "owning" the task. For example, the crowdsourcing system may give each worker a small amount of money for each task he or she completes. [0002] Crowdsourcing systems do not provide direct oversight of the work performed by their workers. The crowdsourcing system may also impose no restrictions (or minimal restrictions) on the workers who are allowed to work on the tasks. As a result, the quality of work performed by different workers may vary. Some workers are diligent and perform high quality responses. Other workers provide lower quality work to varying degrees. Indeed, at one end of the quality spectrum, some workers may correspond to s...

AI Technical Summary

Problems solved by technology

[0003] Among other pitfalls, the presence of low-quality work can quickly deplete the financial resources allocated by the task owner without providing any benefit to the task owner

Images