Computing resource allocation method and device based on hybrid distribution architecture and storage medium

Abstract

The invention provides a distributed computing system based on hybrid computing resources, used for reasonable allocation of resources, to meet the requirement for diversity of computing tasks. The system comprises a computing engine layer and a resource scheduling layer. The computing engine layer is composed of a plurality of deep learning frameworks constructed on the same Spark computing engine, and access interfaces of all the deep learning frameworks are packaged in a unified mode for the computing engine layer. The resource scheduling layer comprises a plurality of heterogeneous computing resources, and the heterogeneous computing resources comprise at least one of a CPU, a GPU and an FPGA. In the resource scheduling layer, different task queues are divided according to the task types of the to-be-processed tasks, different logic clusters are divided according to the types of computing resources carried by different physical machines, and the tasks in the task queues are distributed to the corresponding logic clusters for execution according to the task types of the to-be-processed tasks.

Description

technical field [0001] The present invention relates to the technical field of big data processing, in particular to a computing resource allocation method, device and storage medium based on a hybrid distributed architecture. Background technique [0002] The rise of big data technology has once again inspired the vitality of artificial intelligence. The Go game in 2016 once again ignited the upsurge of artificial intelligence. It is precisely because of the support of mature and stable big data technology that the massive computing tasks behind AlphaGo can be completed. Apache Spark is a fast and general-purpose computing engine designed for large-scale data processing. It has the advantages of Hadoop MapReduce, but unlike MapReduce, the intermediate output results of jobs can be stored in memory, so that there is no need to read and write HDFS (distributed file system), so Spark can be better suited for data mining and machine learning algorithms that require iteration. ...

AI Technical Summary

Problems solved by technology

In recent years, many deep learning frameworks built on Spark have been proposed. Although they are all set up on Spark, the configuration process and user interface are different, and resource preemption will also occur when multiple frameworks are used at the same time.

On the other hand, the core resources of traditional computer computing and processing generally refer to the central processing unit (CPU). However, with the development of artificial intelligence and other technologies, the computing requirements are also diversified, and the pure CPU can no longer meet the real-time performance of computing tasks. Require

Images