Unbalanced data set oversampling method based on genetic algorithm and k-means clustering

Abstract

The invention discloses an unbalanced data set oversampling method based on a genetic algorithm and k-means clustering, and the method comprises the following steps: inputting an original unbalanced data set, and dividing the unbalanced data set into a training data set and a testing data set; dividing the training data set into a positive class sample set and a negative class sample set; clustering the positive class sample set by using a k-means clustering algorithm to obtain a plurality of different clusters; allocating corresponding sampling weights to the number of samples in each cluster; calculating the Mahalanobis distance of the sample data in each cluster, and dividing the sample data into two groups of parent class sample data sets according to the Mahalanobis distance; according to a crossover operator in the genetic algorithm, forming a new positive class sample by by utilizing the parent class sample data set; combining the newly synthesized positive class sample and theoriginal training data set into a balanced data set; training a classifier model by utilizing the balance data set; and evaluating the performance of the classifier model by utilizing the test data set. According to the method, the classification accuracy of the classifier model on the positive samples in the unbalanced data set can be effectively improved.

Description

technical field [0001] The invention belongs to the field of computer data mining, and relates to an oversampling classification method of unbalanced data sets, in particular to an oversampling method of unbalanced data sets based on genetic algorithm and k-means clustering. Background technique [0002] In the field of data classification, there is such a situation: in the data set, the number of samples of different categories varies greatly, among which samples with a large number are called multi-class samples, samples with a small number are called few-class samples, and multi-class samples are called It is called the negative class, and the minority class sample is called the positive class. Usually, Imbalance Rate (IR) is used to measure the degree of imbalance of a data set, which refers to the ratio of the number of negative samples to the number of positive samples in the data set. In many practical application fields, the cost of obtaining positive sample data is...

AI Technical Summary

Problems solved by technology

The more classic sampling algorithms include SMOTE, Borderline-SMOTE, MWMOTE, etc., but these algorithms indiscriminately sample all samples based on the distance of the data samples, without considering the distribution characteristics of the positive samples, resulting in fuzzy overlapping of sample boundaries , it is easy to overfit the model, resulting in lower prediction accuracy

Images