Distributed column subset selection method and system and leukemia gene information mining method

Abstract

The invention discloses a distributed column subset selection method. The method comprises the following steps: acquiring and processing all features in a data set, and uniformly grouping the features to computing nodes; executing a subset quality evaluation method on each computing node and obtaining a corresponding feature subset target feature number; allowing each computing node to perform respective feature selection calculation to obtain features selected by each computing node; and summarizing feature selection calculation results of the calculation nodes to obtain finally selected features. The invention further discloses a system based on the distributed column subset selection method and a leukemia gene information mining method based on the method and the system. According to the method, redundant features are effectively prevented from being selected in a subset, and the feature selection process is accelerated; the selected features are directly summarized as a final selection result, so that the method at least can achieve linear acceleration theoretically; the accuracy is high, the calculation speed is high, and the reliability is better; meanwhile, gene characteristics of leukemia and relevance of leukemia are obtained.

Description

technical field [0001] The invention belongs to the field of big data processing, and in particular relates to a distributed column subset selection method, system and leukemia gene information mining method. Background technique [0002] With the emergence of emerging computer applications such as the Internet of Things, machine learning, computer vision, and natural language processing, people often encounter high-dimensional data with massive numbers of samples and features. Processing these high-dimensional data requires more computing and storage resources, which often cannot be processed by a single machine, and most of the features in these data may be useless and redundant. Therefore, selecting representative features from high-dimensional data and serving computer applications has become an urgent problem to be solved. Therefore, as a method that can effectively select representative features from the original feature set, feature selection technology has become th...

AI Technical Summary

Problems solved by technology

[0009] 3) The two-stage algorithm cannot achieve linear acceleration in theory, so in practice, the calculation of this algorithm is very time-consuming and the practicability is not strong

[0010] Furthermore, the two-stage algorithm assumes that all subsets have the same quality, which is the main reason for the above inadequacies

In fact, the quality between subsets is often different, ignoring this difference will lead to waste of time and resources, and even affect the final result of feature selection

Therefore, the current distributed feature selection method for column subset selection still has the problems of low accuracy, slow calculation speed and poor reliability.

Images