Self-adaptive oversampling method based on HDBSCAN clustering

Abstract

The invention discloses a self-adaptive oversampling method based on HDBSCAN clustering, and mainly solves the problem of unbalanced data classification by using complete data information in an existing method. The technology comprises the following steps: (1) inputting a training data set; (2) clustering the minority class samples in the training set to obtain different scales of clusters which are not intersected with each other; (3) calculating the number of samples needing to be synthesized in each minority class cluster; (4) adaptively synthesizing new samples according to the number of samples needing to be synthesized by each cluster to obtain a new minority class data set; (5) forming a new balanced data set by the majority class data set and the new minority class data set; and (6) training and testing the classifier by using the new balance data set. According to the technology, noise in an unbalanced data set can be effectively prevented from being generated, meanwhile, theproblem of inter-class and intra-class unbalance is solved, and a brand-new oversampling strategy is provided for unbalanced learning.

Description

technical field [0001] The invention belongs to the field of computer artificial intelligence, and in particular relates to an integrated classification method combining unbalanced data resampling technology and clustering. Background technique [0002] Most of the standard machine learning algorithms proposed by researchers in recent years are based on the assumption that the data set distribution is balanced or the error cost is equal. However, in real life, we often encounter a lot of data distributions that are extremely unbalanced. Or scenarios where misclassification costs vary significantly. For example, many current classification learning algorithms are difficult to achieve accurate prediction results in the fields of credit card fraud detection, cancer risk prediction, text classification, software defect prediction, and bioinformatics, because these classification learning algorithms are due to the unbalanced distribution of training data sets, noise, data, etc. ...

AI Technical Summary

Problems solved by technology

Studies have shown that the overall classification performance of classifiers on balanced data is much better than that on original data, so there have been many studies aimed at improving unbalanced learning problems in recent years, but there are currently many methods on the data level. Limitations: For example, random oversampling Random Oversampling is to randomly copy minority class samples to balance the class distribution. This method can effectively improve the classification performance of the classifier but it is easy to lead to overfitting

This method can solve the problem of inter-class and intra-class imbalance at the same time, but it cannot strengthen the classifier's learning of some important minority class samples

[0004] In summary, although most algorithms can overcome some of the shortcomings of existing oversampling algorithms, few algorithms can mitigate inter-class and intra-class imbalance problems while avoiding noise.

In addition, many techniques blindly synthesize new samples, and cannot perform reasonable sampling processing according to the distribution characteristics of the data.

Images