Spark task scheduling method and system based on heterogeneous resources

Abstract

The invention discloses a heterogeneous resource-based Spark task scheduling method, which comprises the following steps that: a server acquires resource information required by a system based on a Linux command and submits the resource information to a resource manager to create a cluster manager and complete initialization, and the server receives a task job submitted by a client, the method comprises the steps that a task job is created, the task job is submitted to a created cluster manager so as to convert the task job into a plurality of RDDs, all the obtained RDDs are analyzed so as to obtain an RDD graph representing the dependency relationship among the RDDs, a server side generates a DAG graph of a scheduling stage according to the dependency relationship among all the RDDs in the RDD graph, and the DAG graph of the scheduling stage is generated according to the dependency relationship among all the RDDs in the RDD graph. And the server divides all the RDDs in the DAG into a first task stage, a second task stage and a third task stage according to the corresponding dependencies. According to the task scheduling system, the performance optimization problem can be solved from all aspects, and the technical problem of system resource waste caused by the fact that a resource manager of an existing task scheduling system is only limited to collection of the number of CPU cores is solved.

Description

technical field [0001] The invention belongs to the field of computer technology, and more particularly, relates to a Spark task scheduling method and system based on heterogeneous resources. Background technique [0002] Task scheduling is an important part of the operating system. For real-time operating systems, task scheduling directly affects its real-time performance. The task scheduling system is a very core component in the data platform. In daily data processing, it is very common to run some businesses regularly, such as importing new data from the database to the data platform regularly, and exporting the data processed by the data platform to the database or file system; so to speak, the task The dispatch system is similar to the commander of the army, directing the operation of each component on the data platform, and supervising the operation of the task at all times. [0003] However, the traditional task scheduling system has some deficiencies that cannot be...

AI Technical Summary

Problems solved by technology

Its purpose is to solve the technical problems of the existing task scheduling system that the resource manager is limited to collecting the number of CPU cores, which leads to waste of system resources, and because the spark task scheduling process involves a large number of RDD data reorganization operations, and performance optimization during the reorganization operation Poor, resulting in greatly reduced operating efficiency, and does not support the technical problems of real-time status update of the resource manager through the operating results to set anomaly detection, and the fact that the data structure for efficient storage is not selected during the execution of the task node, resulting in The technical problem of losing high-efficiency performance, and the technical problem that the running results obtained during the spark task scheduling process will not be updated to the resource manager in real time, so that the running exception cannot be detected

Images