Outlier data mining method based on feature weighting and MapReduce

Abstract

The invention relates to the technical field of data mining, in particular to an outlier data mining method based on feature weighting and MapReduce, which comprises the following steps: step 1, basedon a feature weighting subspace, separating subspace data into a clustering center, a clustering cluster and a candidate outlier data set under a MapReduce programming model; and 2, calculating a global distance for the outlier data set in the step 1, and then defining outlier data. According to the invention, the outlier data mining method is reasonable in calculation amount; human factors are small, and the digging efficiency and precision are high; for high-dimensional mass data, feature dimensions which cannot provide valuable information in the high-dimensional data set are automaticallysearched and deleted, and the interference of dimension disaster is effectively reduced; the invention provides a technical scheme of a high-dimensional massive outlier data mining method which is simple in system, relatively high in accuracy and excellent in performance, so that the efficiency problem in outlier detection is relatively well overcome, and the method has profound application and influence in the field of informationized big data.

Description

technical field [0001] The present invention relates to the technical field of data mining, in particular to a method for outlier data mining Background technique [0002] Outlier data is the data that obviously deviates from other data, does not meet the general pattern or behavior of the data, and is inconsistent with other existing data. It often contains a lot of valuable information that is not easy to be discovered by people. As an important branch of data mining, outlier data mining has been widely used in securities market, astronomical spectrum data analysis, network intrusion, financial fraud, extreme weather analysis and other fields. In high-dimensional mass data, due to the large amount of data and high dimensionality, the effect and efficiency of outlier data mining are seriously affected, and some outlier data hidden in the subspace and some outlier data with edge distribution may not be found. It is precisely because of the clustering characteristics of high...

AI Technical Summary

Problems solved by technology

In high-dimensional mass data, due to the large amount of data and high dimensionality, the effect and efficiency of outlier data mining are seriously affected, and some outlier data hidden in the subspace and some outlier data with edge distribution may not be found.

It is precisely because of the clustering characteristics of high-dimensional sparse data sets that the outlier data distribution often exists in a certain subspace instead of the entire feature space, and irrelevant features will make the cluster structure of the data more blurred. If the cluster structure in the data set is well discovered, the outliers will be more difficult to detect, and outlier data mining cannot be realized.

[0003] In addition, in recent years, although the traditional outlier data mining algorithms have made a lot of improvements in their respective fields, they are no longer applicable in high-dimensional data sets, and the calculation load is large, and the mining efficiency and accuracy are low. Therefore, how to target Accurate mining of big data, high-dimensional data, and outlier data is a major problem in outlier data mining.

Images