Matrix model based multi-calculation-engine automatic selection method

Abstract

The invention discloses a matrix model based multi-calculation-engine automatic selection method. The method comprises the following steps of: adopting a matrix as a programming model and R as a programming language, and calling matrix calculation operations of different calculation engines in an R language program; calculating the time for executing different operations in different engines for matrixes of different scales, and performing analysis modeling to obtain a time performance model; modeling transmission time of data of the matrixes of different scales among the engines to obtain a time performance model; performing grammatical analysis on the R program, constructing a data flow diagram of the matrix operations, and applying each matrix operation to one calculation engine through recursive analysis on the data flow diagram according to a principle of shortest total execution time; performing grammatical analysis on the R language program, and adding set engine information into the R language program to obtain a final execution solution. According to the method, the problems of single platform, low performance and matrix scale change coping capacity shortage of existing matrix calculation are solved.

Description

technical field [0001] The invention relates to the field of automatic scheduling, in particular to an automatic selection method of multiple computing engines based on a matrix model. Background technique [0002] Matrix is ​​an important model in machine learning and data mining. Operations such as matrix multiplication, transposition, and decomposition are necessary methods in many machine learning algorithms. Among the commonly used languages ​​for data analysis and calculation, Matlab and R are based on matrix models, and they perform very fast in small-scale matrix operations. In the era of big data, with the expansion of the matrix scale, there is an upper limit for single-computer computing, and it has become difficult to store and calculate the matrix effectively. With the emergence of distributed computing frameworks such as MPI, Hadoop, and Spark, efficient large-scale matrix operations can be realized by dividing and storing matrix data on each machine in the di...

AI Technical Summary

Problems solved by technology

[0005] Purpose of the invention: In view of the problems and deficiencies in the above-mentioned prior art, the purpose of the present invention is to provide a method for automatically selecting multiple computing engines based on a matrix model, which solves the problem of single, low performance, and large-scale matrices in the existing matrix computing platform. Insufficient ability to change

Images