Concurrent calculation system based on Spark and GPU (Graphics Processing Unit)

Abstract

The invention belongs to the technical field of concurrent calculation and concretely relates to a concurrent calculation frame system based on Spark and a GPU (Graphics Processing Unit). Based on YARN resource management platform, a resource manager and a node manager are improved so that the concurrent calculation frame system can effectively sense GPU resources of a heterogeneous cluster, thereby supporting management and scheduling of the cluster GPU resource; and then in a YARN deployment mode, a job scheduling mechanism and a task execution mechanism of the Spark are improved so that the concurrent calculation frame system supports scheduling and execution for a GPU task. Identification for the GPU resource is introduced in stages such as resource application, resource distribution, DAG (Directed Acyclic Graph) generation, stage division and task execution so that an execution engine can sense the GPU task and effectively executes the GPU task in the heterogeneous cluster; and meanwhile, an effective programming model under the framework is provided by use of the characteristics of efficient memory calculation of Spark in combination with the advantages of multi-core concurrent calculation of the GPU. The system can effectively process data-intensive and compute-intensive jobs and greatly improves the job processing efficiency.

Description

technical field [0001] The invention belongs to the technical field of parallel computing, in particular to a parallel computing framework system based on Spark and GPU. Background technique [0002] In today's society, the scale of data that needs to be processed by various industries has shown a trend of massive amounts, and big data has attracted widespread attention from various industries in society. Undoubtedly, big data contains a wealth of useful information, and if it can be properly mined and used, it will greatly promote scientific research and social economy. Because the information contained in big data can assist business decision-making and scientific research, it has been rapidly developed and applied in many industries. In the era of big data, everything is centered on data, and a lot of effective information that cannot be obtained through other methods can be mined and analyzed from massive historical data, so as to improve the accuracy of decision-making...

AI Technical Summary

Problems solved by technology

Since Spark cannot perceive GPU computing tasks, it cannot distinguish between CPU tasks and GPU tasks. When scheduling task execution, GPU tasks may be started on nodes without GPU devices, resulting in task execution failures.

In addition, in the YARN resource manager, it only supports the scheduling of CPU and memory resources, and cannot perceive GPU resources. It cannot provide the allocation and scheduling of GPU resources to the upper-level Spark framework.

Due to the reasons of YARN and Spark framework itself, the traditional method of performing GPU computing in Spark cannot adapt to the heterogeneous cluster environment

Images