Spark platform based high efficiency text classification method

Abstract

The present invention provides a Spark based high efficiency text classification method. The method comprises: constructing an HDFS file system with a virtual machine and a Spark platform on a physical server, and uploading a data set into the HDFS file system; enabling the Spark platform to read data from the HDFS file system, and converting the data into RDD and storing the RDD into a memory; dividing all tasks into different stages, and then running each task; preprocessing the RDD; performing training; and testing a classification model. The method provided by the present invention makes up the defects of a naive Bayes model and further improves the processing speed; the method also effectively promotes data mining and machine learning and promotes conversion from a conventional data mining algorithm to a parallel data mining algorithm; the method improves classification precision of improving the Bayes algorithm; the method promotes improvement of a Spark platform based algorithm; and finally, the method improves cluster resource utilization.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a high-efficiency text classification method based on the Spark platform. Background technique [0002] With the rapid development of information technology and the gradual widespread use of the Internet, the Internet has now become the most important source of information. Especially with the advent of the era of cloud computing and big data, the data on the Internet is growing exponentially. They have the following characteristics: large amount of data, high dimensionality, complex and irregular structure, and contain a lot of noise data, but they contain a lot of commercial value. Facing such huge and complex information, how to quickly organize, manage, utilize, and dig out valuable information is some very important challenges. [0003] Most of the data today is stored on the Internet in the form of text. Text classification technology is an important basis fo...

AI Technical Summary

Problems solved by technology

[0027] At present, most of the machine learning algorithms are still serial. When the amount of data is not large, serial can be used; but with the advent of cloud computing and the era of big data, the data is growing exponentially, and the traditional serial algorithm obviously cannot meet the requirements. Processing requirements, and the previous grid computing and parallel computing resources utilization rate is not high, resulting in high cost, and requires a dedicated server, ordinary PC can not be used, although Hadoop can meet part of the big data processing, but It realizes the function through the map function and the reduce function, and the communication between the map function and the reduce function is through the HDFS file system

In this way, the number of times Hadoop reads and writes the HDFS file system increases, resulting in an increase in network transmission.

Images