Method and system for classifying data by adopting decision tree

Abstract

The invention discloses a method and system for classifying data by adopting a decision tree. The method comprises the following steps of: parallel computing the information gain of each attribute in training data based on a MapReduce mechanism, and selecting optimum division decision attributes as nodes to construct the decision tree; based on the decision tree, classifying input data records. Aparallel decision tree ID3 algorithm is realized based on the MapReduce. A large data set can be processed, and the parallel efficiency is high. The parallel computing is realized for the nodes in the decision tree and the nodes in the same layer.

Description

technical field [0001] The invention relates to the technical field of data mining, in particular to a data classification method and system using a decision tree. Background technique [0002] Classification is an important topic in data mining. The purpose of classification is to learn a classification function or classification model (also often referred to as a classifier), which can map data items in the database to one of the given categories. Classification can be used to extract models describing important classes of data or to predict future data trends. The purpose of classification is to analyze the input data and find an accurate description or model for each class through the characteristics of the data in the training set. Such descriptions are often expressed using predicates. The resulting class descriptions are used to classify future test data. Although the class labels of these future test data are unknown, we can still predict the class to which these...

AI Technical Summary

Problems solved by technology

[0009] Therefore, many implementations of existing classification decision trees are serial and memory-based, so they cannot handle massive amounts of data; and for existing distributed processing methods, although the data processing scale has been greatly improved, the programming implementation is more complicated and difficult. Difficult, the classification decision tree constructed in addition is not global, but a weighted integration of many local sub-decision trees

More importantly, the efficiency of recursive implementation is relatively low, and the iterative process is uncontrollable

Images