Three-decision unbalanced data oversampling method based on Spark big data platform

Abstract

The invention discloses a three-decision unbalanced data oversampling method based on a Spark big data platform, and relates to a Spark big data technology in the field of data excavation. The method comprises the following steps: firstly, carrying out data transformation with an RDD (Resilient Distributed Dataset) of Spark to obtain a normalized sample set with the LabeledPoint format <label: [features]>, and dividing the sample set into a training set and a test set; secondly, carrying out data variation by adopting the RDD of Spark, calculating a distance between samples, determining the radius of a domain, and classifying the samples in the whole training set into positive domain samples, boundary domain samples and negative domain samples according to a neighborhood three-decision model; then respectively oversampling the boundary domain samples and the negative domain samples; and finally, calling a Spark Mllib machine learning algorithm to verify a sampling result. According to the three-decision unbalanced data oversampling method based on the Spark big data platform, the problem of classification of a large-scale unbalanced data set in the field of machine learning and mode recognition is effectively solved.

Description

technical field [0001] The invention belongs to the fields of data mining, pattern recognition and big data processing, and specifically relates to a three-way decision-making unbalanced data oversampling method based on a Spark big data platform. Background technique [0002] In recent years, mobile phones have already become our daily necessities, and their replacements are quite frequent. It seems that it is becoming more and more common for users to replace their mobile phones. On the one hand, the faster users change their mobile phones, the greater the market value and the higher the manufacturer's income. Therefore, manufacturers need to do everything possible to design new products to stimulate users to replace their mobile phones. On the other hand, major operators are successively using data mining technology to improve marketing efficiency. In actual work, the analysis of customer terminal preferences in the current communication industry is simply based on busine...

AI Technical Summary

Problems solved by technology

Another feature of RDD is that it is elastic. When the memory of the machine overflows during the operation of the job, the RDD will interact with the hard disk data. Although it will reduce efficiency, it can ensure the normal operation of the job.

Images