Active learning classification method based on uncertainty and similarity measurement

Abstract

The invention discloses an active learning classification method based on uncertainty and similarity measurement. The method comprises the following steps: S1, carrying out preprocessing and vectorization on unlabeled classification data; S2, clustering, selecting most representative samples in each class, carrying out manual labeling, recording the samples as a data set L, and recording the rest samples as a set U; S3, calculating a similarity metric value of each sample in the U; S4, enabling the L to be used for training a plurality of different machine learning models, and obtaining the accuracy rate and the output value of each model; S5, determining a weight value and an uncertainty degree of each model so as to determine an uncertainty decision value; S6, determining a diversified training sample with the maximum value, labeling the diversified training sample, updating the labeled diversified training sample to the data set L, and removing the labeled diversified training sample from the set U to obtain an updated set U; and S7, repeating the steps S3-S6 until the accuracy of each model does not change any more, and obtaining a final marked data set L. According to the method, the information redundant sample size can be reduced, and the data labeling cost is reduced on the basis of ensuring the training effect.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to an active learning classification method based on uncertainty and similarity measure. Background technique [0002] Analyzing data in a specific domain can help domain experts mine and discover useful domain knowledge. In the work of a large amount of data-driven modeling, the classification data annotation in the domain is often scarce, the annotation cost is expensive, and the annotator needs to have a strong domain knowledge reserve, which greatly limits the breadth of domain exploration. Aiming at these problems, active learning is currently considered to be a very effective solution. [0003] Active learning is a training data screening method for machine learning that automatically finds this diverse data. Compared with full manual operation, it only takes a fraction of the time to build a better data set and complete data-driven modeling work efficiently. Active learn...

AI Technical Summary

Problems solved by technology

However, the existing active learning methods do not take into account the information overlap problem of the sample for the classifier, that is, the information redundancy problem.

The problem of information redundancy will lead to the selection of redundant meaningless samples and increase the cost of labeling

Images