Method for integrating crowdsourced annotations

Abstract

The present invention provides a method for integrating crowdsourced annotations. According to the method provided by the present invention, a generalized anti-Gaussian distribution is defined by using a regularization hyperparameter, a spacing distance hyperparameter, an annotator voting weight, and a difference between the number of times that annotators annotate a current prediction item as a corresponding estimate and the number of times that annotators annotate the current prediction item as a subcategory; sampling is performed to obtain an auxiliary parameter; and annotator weights are updated by using the auxiliary parameter, thereby significantly enhancing a discrimination capability of a model. Then, a traditional annotation integration majority voting model and a confusion matrix model are integrated, thereby achieving an objective of more comprehensively describing a data generation process. In addition, an accurate prediction item update value is obtained through sampling, and moreover, running efficiency is also improved.

Description

technical field [0001] The invention belongs to the technical field of data mining and machine learning, and more specifically relates to a crowdsourcing labeling integration method. Background technique [0002] With the explosive growth of Internet data and the widespread application of statistical machine learning algorithms, the role of large-scale labeled datasets in machine learning has begun to emerge. At the same time, the method of using crowdsourcing to obtain data annotations is becoming more and more important. Crowdsourcing refers to dividing the workload into a large number of simple subtasks, and then assigning them to a large number of ordinary netizens through the network platform. This mode is currently widely used in scenarios where large-scale datasets such as ImageNet are collected and labeled. Compared with the traditional labeling method, data labeling through crowdsourcing is completed by a large number of ordinary netizens at the same time, which ha...

AI Technical Summary

Problems solved by technology

Compared with the majority voting model, this algorithm adds a lot of details. It has a preliminary assumption about the source of the labeler's error, and gives a more rigorous problem statement through the method of probability. However, there are many far-fetched assumptions in this processing method: first, under this model, the probability that an annotator marks a picture of one category as another category is a certain value, but as the picture changes, , each image may tend to be of a different type, so their probabilities of being classified into the same class may vary greatly

Second, this model does not take into account the tendency of the annotators themselves

[0007] The latest achievements in the above fields provide a good template for the establishment of subsequent models. However, it is still a relatively preliminary algorithm, and there is a lot of room for further improvement.

Images