Computing resource allocation method, device and storage medium based on hybrid distributed architecture

Abstract

The present invention proposes a distributed computing system based on hybrid computing resources to rationally allocate resources and meet the requirements of diverse computing tasks. The system includes a computing engine layer and a resource scheduling layer, wherein: the computing engine layer consists of Composed of multiple deep learning frameworks built on the same Spark computing engine, the access interfaces of each deep learning framework are uniformly encapsulated for the computing engine layer; the resource scheduling layer includes a variety of heterogeneous computing resources, and the heterogeneous computing The resources include at least one of the following: CPU, GPU and FPGA; in the resource scheduling layer, different task queues are divided according to the task types of tasks to be processed, and different logical clusters are divided according to the types of computing resources carried by different physical machines. Task class for processing tasks, assigning tasks in the task queue to corresponding logical clusters for execution.

Description

technical field [0001] The present invention relates to the technical field of big data processing, and in particular, to a computing resource allocation method, device and storage medium based on a hybrid distribution architecture. Background technique [0002] The rise of big data technology has once again stimulated the vitality of artificial intelligence. The Go battle in 2016 once again detonated the upsurge of artificial intelligence. It is precisely because of the mature and stable big data technology support that the massive computing tasks behind AlphaGo can be completed. Apache Spark is a fast and general computing engine designed for large-scale data processing. It has the advantages of Hadoop MapReduce. However, unlike MapReduce, the intermediate output results of Job can be stored in memory, so there is no need to read and write HDFS. (distributed file system), so Spark is better suited for iterative algorithms such as data mining and machine learning. In rece...

AI Technical Summary

Problems solved by technology

In recent years, many deep learning frameworks built on Spark have been proposed. Although they are all set up on Spark, the configuration process and user interface are different, and resource preemption will also occur when multiple frameworks are used at the same time.

On the other hand, the core resources of traditional computer computing and processing generally refer to the central processing unit (CPU). However, with the development of artificial intelligence and other technologies, the computing requirements are also diversified, and the pure CPU can no longer meet the real-time performance of computing tasks. Require

Images