Unbalanced data processing method for synthesizing minority class samples based on relationship between features

Abstract

The invention relates to an unbalanced data processing method for synthesizing minority class samples based on relationships between features. The method comprises the steps that firstly, when minority class data are sampled, for high-dimensional data with redundancy characteristics, the redundancy characteristics can influence the performance of a sampling algorithm, a multi-target-based pareto leading edge characteristic method is introduced, three evaluation indexes including an AUC value, an Fscore value and a Gmean value are used as fitness functions, and the performance of a sampling algorithm is improved through continuous iterative optimization; the pareto leading edge features with the three high evaluation indexes are selected; then, performing feature sampling on pareto frontier features of a minority class of samples, using XGBoost regression to capture a relationship between the features, considering a quality problem of a new sample while generating the sample, and designing an evaluation index DIS for calculating sample quality based on Euclidean distance in order to evaluate the quality of the new sample. And finally, newly generated samples are added into an original data set, a plurality of balanced sample sets are synthesized, and a final classification result is output by using majority voting integration.

Description

technical field [0001] The invention relates to the technical fields of machine learning and data mining, in particular to an unbalanced data processing method for synthesizing minority class samples based on the relationship between features. Background technique [0002] With the continuous expansion of data availability in many large-scale, complex and networked systems such as medical care, security, Internet and finance, how to intelligently process data and extract valuable information from data has become a research hotspot in theory and application. Although existing knowledge discovery and data engineering techniques have achieved great success in many real-world applications, the problem of learning from unbalanced data is still a relatively difficult challenge, which is a common problem in different life domains. A common phenomenon, misclassification of minority classes is extremely costly, and has attracted increasing attention from both academia and industry. ...

AI Technical Summary

Problems solved by technology

However, in most cases, when oversampling or undersampling, the relationship between the underlying data features is not considered. Oversampling may lead to overfitting problems, while undersampling may discard samples with large amounts of information.

Images