Data resampling method based on clustering oversampling and instance hardness threshold

Abstract

The invention provides a data resampling method based on clustering oversampling and an instance hardness threshold. The method comprises the following steps: firstly, performing clustering processingon a data set by utilizing a Kmeans method, and performing filtering processing and sampling weight distribution on clustering; then, adopting an SMOTE algorithm to carry out oversampling on the dataset to generate new data, so that the number of minority class samples in the data set is equal to that of majority class samples, and the data set becomes class balance; and finally, cleaning the data by adopting an instance hardness threshold algorithm to obtain a final balanced data set with fewer noisy points. According to the method, the class imbalance data set can be processed into the balance data set, and the prediction performance of the classifier for minority class samples is improved.

Description

technical field [0001] The invention belongs to the technical field of unbalanced data processing, and in particular relates to a data resampling method based on clustering oversampling and instance hardness threshold. Background technique [0002] In machine learning, since the goal of the classification algorithm is to improve the classification accuracy, when there is a class imbalance problem in the data set, the classification algorithm will tend to predict the sample as the majority class. This is because even if the classification algorithm cannot correctly classify the minority class samples, it can still achieve a classification result with a high accuracy rate. Although the number of minority class samples is small, the importance is higher, and the cost of misclassifying minority class samples is much higher than the cost of misclassifying majority class samples. Therefore, addressing the class imbalance problem is very important for model predictions on imbalanc...

AI Technical Summary

Problems solved by technology

[0003] In order to solve the problem of prediction deviation of classification algorithms for minority samples in unbalanced text data, the present invention proposes a data resampling method based on clustering oversampling and instance hardness threshold

Images