Unbalanced data classification method based on adaptive upsampling

Abstract

The invention relates to an unbalanced data classification method based on adaptive upsampling. The method includes the following steps of calculating the total of positive samples to be newly generated; calculating the probability density distribution for each positive sample by taking the Euclidean distance as the metric; determining the number of the new samples to be generated of the positive sample; generating a new positive sample and adding the newly generated positive sample points to an original unbalanced training set to make the positive and negative samples be same in number, namely, obtaining a new balance training set including n<n> positive samples and n<n> negative samples; and training the newly generated balance training set by means of an Adaboost algorithm and obtaining a final classification model after the iteration for T times. According to the invention, the classification performance of the unbalanced dataset is improved.

Description

Technical field [0001] The invention relates to pattern recognition technology, in particular to a classifier for unbalanced data sets. Background technique [0002] With the rapid development of data mining, pattern recognition and machine learning technologies, data classification has been applied and played an important role in many fields such as image retrieval, medical detection and diagnosis, polygraph detection, text classification and crude oil spill detection. However, classical classification algorithms such as support vector machines, artificial neural networks, and linear discriminant analysis are designed with the assumption that the training data set contains approximately the same number of samples for each class. But in fact, in the above-mentioned fields, the number of abnormal samples (positive samples) is often much less than that of normal samples (negative samples). At this time, in order to obtain a higher overall accuracy, the classical classifier wi...

AI Technical Summary

Problems solved by technology

However, since the traditional Adaboost algorithm itself does not pay too much attention to positive samples, the effect is still not ideal

[0006] From the above analysis, it can be seen that although both the data-level method and the algorithm-level method can reduce the impact of data imbalance on the classification effect, both methods have certain limitations.

Images