Data dimension reduction method based on parallel principal component analysis (PCA) algorithm

Abstract

The invention discloses a data dimension reduction method based on a parallel principal component analysis (PCA) algorithm. The method comprises the steps of: S1, constructing a sample data matrix D<nxm> by high-dimensional data of which dimensions are to be reduced; S2, calculating a covariance matrix C<mxm> of the sample data matrix D<nxm>; S3, calculating m feature values of the covariance matrix C<mxm> and m corresponding feature vectors; S4, determining the number k of principal components according to the feature values and the feature vectors; and S5, utilizing the feature vectors, which correspond to the top-k greater feature values, to construct a transformation matrix, and utilizing the transformation matrix to calculate a principal component matrix, wherein the principal component matrix is data of which the dimensions are reduced. According to the method, the problem that according to a traditional stand-alone principal component analysis algorithm, the data cannot be loaded into a memory at once because a data size is too large is overcome, I / O operations are reduced, and the processing efficiency of data dimension reduction is improved.

Description

technical field [0001] The invention relates to a high-dimensional data linear dimensionality reduction technology, in particular to a data dimensionality reduction method based on a principal component analysis algorithm. Background technique [0002] With the continuous development of network information technology and mobile Internet, the amount of data in different business vertical fields of enterprises is increasing. How to discover valuable information from these data and provide important decision-making support for enterprises has become the key to the success of enterprises. These data often have two characteristics: one is the large scale of the data; the other is the high dimensionality of the data. Large-scale high-dimensional data poses challenges to data transmission, storage, and data pattern discovery. How to efficiently process large-scale high-dimensional data and effectively pattern discovery is particularly important. In these high-dimensional data, the...

AI Technical Summary

Problems solved by technology

[0005] The present invention aims at the deficiencies in the prior art, and provides a principal component analysis method using the MapReduce parallel computing framework, which solves the problem that the traditional stand-alone principal component analysis algorithm cannot be loaded into the memory at one time because the data scale is too large, and is conducive to reducing I / O operation to improve the processing efficiency of data dimensionality reduction

Images