Data selection method and device

Abstract

The invention discloses a data selection method and device, relates to the technical field of data processing, and mainly aims at optimizing sample selection in an active learning process and avoiding the problem of selection inclination of sample data. The main technical scheme of the invention is as follows: obtaining a candidate sample set, wherein the candidate sample set comprises a plurality of candidate samples belonging to different categories; according to the candidate samples belonging to the same category, calculating the uncertainty of the category; according to the uncertainty of different categories, calculating category uncertainty distribution of the candidate sample set; and selecting a first candidate sample in a first category to enter a sample pool according to the category uncertainty distribution of the candidate sample set.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a data selection method and device. Background technique [0002] With the popularity of artificial intelligence, deep learning has made great breakthroughs in various practical applications. Many problems are no longer limited, but obtaining a large amount of accurately labeled data still requires high costs, and model training still requires a lot of time. And energy, these have also become the limitations of current deep learning. Active learning can use fewer labeled samples to achieve higher model learning accuracy by screening unlabeled data. [0003] Active learning is a subfield of artificial intelligence, also known as query learning and optimal experimental design in the field of statistics. The algorithm consists of two basic modules: a learning module and a selection strategy. Active learning uses the "selection strategy" to actively select some samples fro...

AI Technical Summary

Problems solved by technology

However, the existing active learning mainly considers the uncertainty of the sample itself (that is, the degree to which the sample cannot be effectively identified and distinguished by the model) and the correlation between the sample and other samples (that is, the sample The degree of approximation between them), when it is necessary to select a large number of samples of different categories, if the number of samples of a certain type and the number of uncertainties are large, the sample selected through active learning tends to be similar to that type of sample, resulting in the sample data Tilt, which affects the optimization effect of model training

Images