Concurrent calculation system based on Spark and GPU (Graphics Processing Unit)

A parallel computing and distributed computing technology, applied in computing, computers, digital computer components, etc., can solve problems such as inability to perceive GPU resources, inability to distinguish between CPU tasks and GPU tasks, and task execution failures
CN107168782AInactive Publication Date: 2017-09-15FUDAN UNIV
4 Cites 36 Cited by

Patent Information

Authority / Receiving Office
CN Β· China
Current Assignee / Owner
FUDAN UNIV
Publication Date
2017-09-15
Estimated Expiration
Not applicable Β· inactive patent

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention belongs to the technical field of concurrent calculation and concretely relates to a concurrent calculation frame system based on Spark and a GPU (Graphics Processing Unit). Based on YARN resource management platform, a resource manager and a node manager are improved so that the concurrent calculation frame system can effectively sense GPU resources of a heterogeneous cluster, thereby supporting management and scheduling of the cluster GPU resource; and then in a YARN deployment mode, a job scheduling mechanism and a task execution mechanism of the Spark are improved so that the concurrent calculation frame system supports scheduling and execution for a GPU task. Identification for the GPU resource is introduced in stages such as resource application, resource distribution, DAG (Directed Acyclic Graph) generation, stage division and task execution so that an execution engine can sense the GPU task and effectively executes the GPU task in the heterogeneous cluster; and meanwhile, an effective programming model under the framework is provided by use of the characteristics of efficient memory calculation of Spark in combination with the advantages of multi-core concurrent calculation of the GPU. The system can effectively process data-intensive and compute-intensive jobs and greatly improves the job processing efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention belongs to the technical field of parallel computing, in particular to a parallel computing framework system based on Spark and GPU. Background technique

[0002] In today's society, the scale of data that needs to be processed by various industries has shown a trend of massive amounts, and big data has attracted widespread attention from various industries in society. Undoubtedly, big data contains a wealth of useful information, and if it can be properly mined and used, it will greatly promote scientific research and social economy. Because the information contained in big data can assist business decision-making and scientific research, it has been rapidly developed and applied in many industries. In the era of big data, everything is centered on data, and a lot of effective information that cannot be obtained through other methods can be mined and analyzed from massive historical data, so as to improve the accuracy of decision-making...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More