Unbalanced data set preprocessing method based on neighborhood information

Abstract

The invention provides an unbalanced data set preprocessing method based on neighborhood information. The method comprises the following steps of S1, randomly selecting a minority class sample in a target unbalanced data set; S2, constructing a neighborhood of a spatial hypersphere shape with the selected minority class sample as the center; S3, judging whether all constructed neighborhoods contain all minority class samples or not, if yes, executing the step S4, and if not, returning to execute the step S1; S4, determining weights of all neighborhoods according to the number of minority class samples contained in the neighborhoods; S5, determining the number of new samples needing to be synthesized in each neighborhood; and S6, synthesizing a new sample for each neighborhood, according to the preprocessing method, the synthesized new sample region is constrained through detection in geometric space, different sampling multiplying powers are provided for the samples with different subsequent classification contribution degrees through a weighted synthesis strategy, identification and classification of a traditional algorithm are more convenient, and the precision is higher.

Description

technical field [0001] The invention relates to the technical field of data mining and data preprocessing, in particular to a method for preprocessing unbalanced data sets based on neighborhood information. Background technique [0002] With the rapid development of the mobile Internet of Things in the information age, the amount of data in various industries and fields has also shown explosive growth. How to identify a small amount of truly meaningful data in the massive data has become a problem in the field of data mining in the field of machine learning. Hotspots and Difficulties. [0003] Data identification and classification have been widely used in real life, and many mature classification algorithms have emerged. For balanced data sets, most of these traditional algorithms have a good classification result, and in practical applications, very Most datasets are unbalanced datasets. The number of samples of a certain type in a dataset is significantly less than the n...

AI Technical Summary

Problems solved by technology

[0003] Data identification and classification have been widely used in real life, and many mature classification algorithms have emerged. For balanced data sets, most of these traditional algorithms have a good classification result, and in practical applications, very Most datasets are unbalanced datasets. The number of samples of a certain type in a dataset is significantly less than the number of other samples. This kind of dataset is called an unbalanced dataset, such as common medical diagnosis datasets in life, For network identification data sets, harassment and interception data sets, etc., the traditional classification method is more inclined to the overall classification accuracy rather than the minority class data. When the distribution of the data set is uneven and the minority class overlaps with the majority class, the traditional classification algorithm will The samples at the edge of the classification hyperplane are directly classified into the majority class samples to improve the overall classification accuracy. At the same time, the recall rate of the minority class is greatly reduced. Although the minority class data has little impact on the overall accuracy, it is extremely important. For example, a There are 1000 samples in the data set, 999 majority classes and 1 minority class, then all the data is divided into majority classes by traditional classification methods, then the overall classification accuracy can reach 99.9%, but for the minority class, the classification 100% error rate

With the popularity of its learning in real life, people have higher requirements for the recognition rate of minority classes in unbalanced datasets. At the same time, traditional classification methods have various defects in the identification and classification of unbalanced datasets. cannot provide satisfactory classification results, therefore, the research on classification methods for imbalanced datasets is an urgent problem to be solved at present, and has important theoretical and practical significance

Images