Spark memory computing big data platform-based CLR multi-label data classification method

Abstract

The invention provides a Spark big data platform-based CLR multi-label classification method, and relates to a data mining technology. Each data set is divided into an area through a relationship between characteristics and labels; one part of data sets are randomly extracted from the data sets in a training set as test sets; base classifiers are built by the remaining data sets as the training set; and the test sets are used for testing a classifier and modifying parameters of the base classifiers thereof, thereby selecting the best base classifier from multiple trained base classifiers as the base classifier of the data set. A prediction set is predicted by using the final base classifiers. A CLR multi-label learning algorithm is combined with a memory-based efficient computing theory of Spark, no correlation after label transformation in a CLR algorithm is fully utilized, the interference among different base classifiers is reduced, the operating speed of a computing framework of the Spark is fully utilized and data can be effectively mined.

Description

technical field [0001] The invention relates to computer information processing and data mining technical fields, and provides a multi-label data mining method based on a calibration label ranking algorithm CLR (Calibrated Label Ranking) of a Spark big data platform. Background technique [0002] With the development of information technology, Internet data and resources are characterized by massive quantities. In order to effectively manage and utilize these massive amounts of information, content-based information retrieval and data mining have gradually become areas of concern. While the amount of data is increasing, the complexity of the data labeling structure is also increasing. The traditional single-label data mining can no longer meet the needs of technological development. The importance of multi-label data mining is gradually highlighted. The technology involved Applications are also increasing, such as semantic annotation of images and videos, gene functional gr...

AI Technical Summary

Problems solved by technology

[0012] The present invention aims at the defects in the prior art that a large amount of historical data cannot be effectively used in a timely manner after the data is acquired and beneficial information can be quickly excavated from it, data processing cannot be performed quickly, and model building takes a lot of time.

Images