Data processing method and device

Abstract

The invention discloses a data processing method and device and relates to the technical field of distributed computation. Computational node mixed establishment distributed computation systems of multiple processor types can be adopted, for example, computational nodes and the like CPU processors are adopted in the whole computation system in a mixed mode, preset corresponding types of vector templates are adopted to load input data of a user according to the data types, then before computation vectors are split into computation sub-vectors to be distributed to the computational nodes for computation, OP operators edited by the user are firstly split into combinations of sub-operators, the computational nodes matched with the processor types during computation of the computational nodes are judged, and then data is sent to corresponding computational nodes for computation. A computation platform is achieved in a heterogeneous mode, the computational nodes of different processor types are used in parallel, and the universality and compatibility are improved.

Description

technical field [0001] The invention relates to the technical field of distributed computing, in particular to a data processing method and device. Background technique [0002] As big data analysis research has become a hot topic, MPI-based distributed memory computing platforms have once again attracted attention in the industry in recent years. [0003] For the analysis and processing of massive big data, the popular distributed computing platforms generally include the following types: MapReduce computing in hadoop, spark, streaming computing (represented by storm), and memory computing based on mpi. But the bottom layer of the hardware of most of these platforms adopts CPU as the core computing node. Using a platform that uses CPU computing nodes at the bottom layer may cause the problem of slow computing speed due to CPU performance limitations. [0004] In addition, some professional fields have recently launched some dedicated platforms, but the computing nodes use...

AI Technical Summary

Problems solved by technology

Using a platform that uses CPU computing nodes at the bottom layer may cause the problem of slow computing speed due to CPU performance limitations

[0004] In addition, some professional fields have recently launched some dedicated platforms, but the computing nodes used are also a single model. In CPU (Central Processing Unit, central processing unit), GPU (Graphic Processing Unit, graphics processing unit), FPGA (Field-Programmable Gate Array, field programmable gate array), DSP (digital signal processor, digital signal processor), and only play a role in a certain professional field, lack of support for versatility and compatibility, once the business model changes , these specialized computing platforms are often unusable or unable to take advantage of

[0005] In short, the use of a single computing unit cannot give full play to the performance advantages for different computing requirements, and it may also have insufficient versatility and compatibility support.

Images