Large-scale data mining method capable of guaranteeing quality monotony

Abstract

The invention provides a data mining method capable of guaranteeing quality monotony. The method comprises the following steps: after an original big data set is compressed by a PCA (principal components analysis) technology, mapping the original big data set onto an R-tree data structure; then, carrying out mining processing on the data set by an improved K-nearest neighbor classification algorithm. The method mainly comprises the following two parts including a coding part and a mining part, wherein the coding part utilizes R-tree to present data, data with high similarity in the data is combined to serve as one node of the R-tree so as to achieve a purpose of mass data compression and improve the efficiency of the mining part; the mining part utilizes the thought of the improved K-nearest neighbor classification algorithm to process the data node and predict the classification of an input test point. According to the large-scale data mining method, the problem that the quality of a mining result and resource restriction cannot be balanced and the quality monotony of an approximate result cannot be guaranteed when big data is mined by a traditional algorithm under the restriction of limited time and resource restriction can be solved.

Description

Technical field [0001] The invention relates to a method for efficiently processing data, by which the monotonicity of the quality of large-scale data mining results is guaranteed, and belongs to the cross-technology application field of data mining, big data and computer software. Background technique [0002] The data capacity and type diversity of big data make us willing to use algorithms to produce approximate results for data mining of big data. Traditional algorithms are difficult to mine big data under limited time and resource constraints. The balance between the quality and resource constraints and the monotonicity of the quality of the approximate results. To solve this problem, we designed a big data mining method that guarantees monotonic quality based on Shannon entropy. The mining method is divided into two parts: the coding part and the mining part. By ensuring the monotonicity of the entropy of the algorithm coding part and the mining part The entropy preservati...

AI Technical Summary

Problems solved by technology

[0002] The data capacity and type diversity of big data make us willing to use algorithms to generate approximate results to mine big data. Traditional algorithms are difficult to achieve mining results when min

Images