Spark-based parallel random label subset multi-label text classification method

Abstract

The invention provides a parallel random label subset multi-label text classification method for Spark-based big data platform. First of all, large scale text data sets and configuration information files are read, the distributed data set RDD is created, the training data set and the prediction data set are cached in memory to complete the initialization operation. Secondly, the label subset of the required numbers is randomly generated in parallel, a new training set is generated for each label subset by the original training set, once again, multiple tags of the new training set through the tag power set are converted into single labels, the data sets are converted into single label multiple data sets, and a base classifier is trained in parallel for these data sets. Then the single label multiple prediction results obtained by prediction are converted into multi-label results. Finally, all the predicted results are collected and voted, to obtain the final multi-label prediction results of the test set. The multi-label text classification method improves the classification accuracy, dramatically reduces the learning time of handling large scale multi-label data.

Description

technical field [0001] The invention relates to the fields of information technology, cloud computing, data mining, text classification, etc., and provides a parallelized random label subset multi-label text classification algorithm based on the Spark big data platform. Background technique [0002] With the development of information technology, the scale of Internet data has grown massively, and the forms of expression have also been continuously enriched. Text is an important information carrier. The development of its automatic classification technology can improve the processing efficiency of massive information, save processing time, and facilitate the use of users. It has received extensive attention and rapid development in recent years. Traditional supervised learning believes that each sample has only one label, and lacks the ability to accurately express the complex semantic information of things. However, a sample may correspond to multiple labels related to it....

AI Technical Summary

Problems solved by technology

[0008] The present invention aims at the shortcomings of the existing multi-label classification technology, such as easy memory overflow, long time and failure to run downtime when class

Images