Semi-supervised random forests classification method based on Spark

Abstract

The invention discloses a semi-supervised random forest classification method based on Spark, which utilizes a random forests algorithm to employ replacement sampling on a training data set and column attributes, so that randomness is added in both row and column directions to ensure decision-making tree diversity, and to avoid tree pruning; in addition, a category is determined in a voting method, and accuracy is greatly improved. Accordingly, the random forests algorithm does not need to perform dimensionality reduction in processing high-dimensional data samples, and has sound effects for a sparse vector as well as a dense vector. According to the verification of a plurality of sets of experiments, the semi-supervised learning algorithm reduces a classification model error rate mean value, and improves calculating performance.

Description

technical field [0001] The invention relates to the technical field of machine learning classification algorithms, in particular to a spark general parallel framework, Tri-Training semi-supervised classification and distributed random forest classification algorithms. Background technique [0002] Classification is a category in the field of machine learning and is a commonly used data analysis technique, which is mainly used to determine which predefined target class a certain target object belongs to. It has a wide range of applications in many fields, including data mining, statistical learning, neural networks, etc. In the field of machine learning, there are two commonly used research methods, one is supervised learning and the other is unsupervised learning. Traditional classification models need to be trained on the basis of labeled sample sets, which belongs to supervised learning. In practical applications, the labeling of data is limited by various factors, and s...

AI Technical Summary

Problems solved by technology

In order to solve the problems of high cost of manual labeling samples, too large sample data cannot be executed quickly on a single machine, too few training samples cannot obtain a classification model with high classification accuracy, and cannot use a large amount of cheap unlabeled data, etc.

Images