A computing framework selection method and device

Abstract

An embodiment of the present invention provides a computing framework selection method, including: selecting a preset component according to a predetermined data mining process; converting the preset component into a directed acyclic graph; selecting a computing framework for data mining for a node; submitting the node to the data mining computing framework, and use the data mining computing framework to calculate the nodes; wherein, the nodes and components of the directed acyclic graph have a corresponding relationship. The embodiment of the present invention also provides a calculation frame selection device.

Description

technical field [0001] The invention relates to the field of data mining, in particular to a calculation framework selection method and device. Background technique [0002] With the advent of the information age, the accumulation of data has grown exponentially. In order to be able to process the existing massive data, various parallel computing framework data processing platforms have emerged. There are mainly two existing parallel computing frameworks: Mapreduce and Spark. Among them, Mapreduce is an offline computing framework. Compared with park, it is also a disk computing framework. When this framework is used for computing, it abstracts an algorithm into two stages of Map and Reduce, which is very suitable for data-intensive Computing; Spark is a memory computing framework that puts data in memory as much as possible to improve the computing efficiency of iterative applications and interactive applications. [0003] At present, big data mining platforms are all imp...

AI Technical Summary

Problems solved by technology

[0003] At present, big data mining platforms are implemented by choosing one of the computing frameworks. If the Mapreduce framework is selected, when the amount of data processed is relatively small compared to the amount of resources in the cluster, the processing speed will be very slow compared with Spark, which is a serious waste. The resources of the cluster are reduced, and the utilization rate of resources is reduced; when the Spark framework is selected, when the amount of data processed is much larger than the amount of resources of the cluster, the resources of the cluster will be seriously insufficient, the performance of processing data will drop sharply, and even the data cannot be processed.

Images