Method for integrating crowdsource annotation data based on task difficulty and annotator ability

Abstract

The invention discloses a method for integrating crowdsource annotation data based on task difficulty and annotator ability. The method involves the following two phenomena: (1) the annotation results of most of the tasks annotated by an annotator with relatively high ability is the same as those of other annotators; (2) the consistency of the annotation results of the tasks with relatively low difficulty and annotated by the annotators is high. According to the method, a novel evaluation method for the task difficulty and an evaluation method for the annotator ability are provided; an integrating method based on the two methods for the crowdsource annotation data is created; the iteration mode is utilized to fast solve; therefore, the ability evaluation of the annotator can be objective and accurate; the difficulty of various crowdsource annotation tasks can be effectively evaluated conveniently; meanwhile, the method is applicable to crowdsource annotation data of various models, including but not being limited to two-value annotation and multi-value annotation of images, texts, videos and other tasks.

Description

technical field [0001] The invention belongs to the technical field of data labeling, and in particular relates to a crowdsourcing labeling data integration method based on task difficulty and labeler's ability. Background technique [0002] High-quality labeled datasets are very important resources in the field of computer research and applications. Algorithms in the fields of computer vision, artificial intelligence, and machine learning are mostly trained and optimized based on corresponding labeled data sets. Obtaining high-quality and large-scale labeled datasets quickly and efficiently has always been a concern of various researchers. The traditional way to obtain labeled datasets is to hire experts to manually label the datasets. The annotation data obtained in this way is of high quality, but the annotation takes a long time, and the financial cost of hiring experts is also very large. [0003] In recent years, with the development of crowdsourcing technology, the...

AI Technical Summary

Problems solved by technology

Although the above method improves the integration accuracy of labeled data to a certain extent, in the process, the definition of the ability of the labeler is only based on the consistency between all the data marked by the labeler and the final labeling results of each task. to determine

However, the tagging results of tasks obtained through integration are not necessarily correct, which leads to biased assessment of the capabilities of the taggers. Therefore, the final tagging obtained in the above-mentioned integration model based on worker capabilities There is also a large deviation in the accuracy of the results

At the same time, the current labeling data integration model lacks the evaluation method of task difficulty, an important influencing factor, and then ignores the important role of task difficulty in the entire process of labeling data integration, resulting in a large deviation in the final labeling results.

Images