Distributed vector computing frame

Abstract

The invention relates to a distributed vector computing frame which comprises an MPI frame and an MPI interface layer used for being connected with the MPI frame. A user oriented computing vector layer is arranged on the MPI interface layer, and comprises a distributed vector interface layer and a MapReduce interface layer. The distributed vector computing frame effectively combines flexible and efficiency characteristics of a MapReduce module and an MPI parallel model, and the flexibility and the expandability of the frame are improved. The matrix and vector data structure characteristics of the computing process of a PageRank algorithm and an abstract algorithm are optimized on the basis of the distributed vector computing frame, and the distributed computing efficiency is effectively improved through a distributed vector interface. Through the optimization application of the distributed vector computing frame in a machine learning typical algorithm PageRank, the usability of the distributed vector computing frame is fully verified, and a support is provided for further improving the distributed computing application service of machine learning algorithms.

Description

technical field [0001] The invention relates to a computing framework, in particular to a distributed vector computing framework, which belongs to the technical field of distributed computing. Background technique [0002] At present, MPI is generally used in the solution to realize large-scale and high-performance machine learning algorithms directly, but researchers need to be familiar with MPI programming, and MPI programming is more biased towards the bottom layer, and the bottom layer knowledge that needs to be paid attention to in the process of programming too complicated. Moreover, in the research process, it is also necessary to consider the division of data between different nodes, the control of communication between nodes, the synchronization mechanism between nodes and the fault tolerance mechanism in the calculation process, etc. Therefore, the development cycle is long and the code reuse degree is low. Using MPI for development will have certain complexity fo...

AI Technical Summary

Problems solved by technology

Therefore, the development cycle is long and the code reuse is low. Using MPI for development will have a certain complexity for the parallelization of the algorithm.

[0003] Machine learning algorithm solutions lack a framework similar to MapReduce (a data parallel model proposed by Google). The MapReduce framework is widely used in the field of distributed computing. Users only need to pay attention to the upper-level logic of data processing, and can develop distributed The underlying data division, node data synchronization and other mechanisms are transparent to upper-level users, and the code is highly reusable, but the support for distributed vector calculation is not enough, and the development of distributed high-performance computing will be too complicated

Images