GPU cluster-based multidimensional big data factorization method

Abstract

The invention discloses a GPU cluster-based multidimensional big data factorization method, aims to solve the problem that a conventional grid parallel factor analysis model cannot process large-scale and high-dimension multidimensional data analysis, and provides an effective pattern processing unit-based multidimensional big data multi-mode decomposition method, namely a hierarchical parallel factor analysis framework. The framework is based on the conventional grid parallel factor analysis model, comprises a process of integrating tensor subsets under a coarse-grained model and a process of calculating all of the tensor subsets and fusing factor subsets under a fine-grained model, and is operated on a cluster formed by a plurality of nodes; each node comprises a plurality of pattern processing units. Tensor decomposition on pattern processing unit equipment can fully utilize a powerful parallel computing capability and a paralleling resource generated in tensor decomposition; experimental results show that through the adoption of the method, executive time for acquiring tensor factors can be greatly shortened, the large-scale data processing capability is improved, and the problem that the computing resource is insufficient is well solved.

Description

technical field [0001] The invention belongs to the technical field of signal analysis, and relates to a multidimensional big data analysis method, in particular to an efficient multidimensional big data factorization method based on a GPU cluster. Background technique [0002] In complex applications based on data analysis, it is necessary to reflect the dynamic characteristics of large-scale tensors during the decomposition process without causing large data deformation. Today, the data scale and data dimension are constantly increasing. Today, it is facing more and more challenges. Finding useful information of data from multidimensional data is becoming more and more important in today's science and engineering, such as feature extraction and dimensionality reduction. Two-dimensional data decomposition methods, such as singular value decomposition (SVD), principal component analysis (PCA), and independent component analysis (ICA), directly applying these methods to the d...

AI Technical Summary

Problems solved by technology

[0004] PARAFAC can analyze data of any size and dimension, but its computational complexity is high, and the performance requirements of the computer are also high. Therefore, the sliding window is used to divide the entire data, and PARAFAC is used to analyze the data in a certain dimension one by one. This lays the foundation for the dynamic tensor analysis of big data. Under this theory, the direct fusion of data makes the correlation between the data partly lost, and the obtained causal joint factor is difficult to reflect the dynamic characteristics of the original data.

[0005] In order to solve the problem of large-scale data, the mathematical theory of PARAFAC has some innovations. It regards large-scale data as the grid of small data, that is, gridPARAFAC ([Document 4]), which converts the decomposition of tensors into independent tensors. The decomposition of the quantum set, the output of the fusion tensor subset results can get all the factors of the tensor, this method is effective, but it faces two major problems, insufficient computing resources and the result differentiation caused by the division of the tensor subset

Images