Parallel feature selection method based on MapReduce

Abstract

The invention discloses a parallel feature selection method based on a MapReduce. The method includes a, dividing data into parts; b, distributing the data parts to a Map computing node; c, establishing a selected feature set and a feature set not selected; d, figuring out combined mutual information between vectors and class variables; e, allowing a maximum feature variable of the combined mutual information to serve as a feature vector, and transmitting a feature variable serial number and corresponding combined mutual information values to a Reduce computing node; f, judging whether a feature vector which is equal to the count exists or not; g, comparing the combined mutual information values; h, adding and deleting selected vectors; i, judging the numbers of the selected features. According to the parallel feature selection method based on the MapReduce, a feature vector combination with the largest amount of information can be extracted rapidly and effectively from massive data sets; the method is adaptable to feature extraction of massive data.

Description

technical field [0001] The present invention relates to a parallel feature selection method based on MapReduce, more specifically, to a parallel feature selection method based on MapReduce that can perform fast feature extraction on massive data. Background technique [0002] In recent years, datasets in many fields, such as genome engineering, text classification, image retrieval, and customer relationship management, have become larger and larger both in terms of quantity and the number of feature variables. This can pose serious problems for the scalability and learning performance of many machine learning methods. How to select the most informative combination of feature variables is a crucial issue. Feature selection is the process of selecting a feature subset from the original feature set according to certain evaluation criteria, thereby effectively reducing the dimensionality of the feature variable set. Feature selection can effectively reduce the dimensionality o...

AI Technical Summary

Problems solved by technology

Conventional feature selection methods cannot handle large-scale datasets

Images