Distributed task scheduling method and system

Abstract

The embodiment of the invention discloses a distributed task scheduling method and system. The method is executed by a working node included in a distributed scheduling system, the distributed scheduling system comprises a plurality of main nodes, and the method comprises the following steps: periodically sending a parameter acquisition request to at least one main node, and receiving a task typeand a total task load fed back by the main node for local storage; when a new task processing condition is detected, calculating a request task number according to the number of tasks in processing and the total task load, and constructing a task acquisition request according to the request task number and the task type; and sending a task acquisition request to a first target main node determinedin the plurality of main nodes, wherein the task acquisition request is used for indicating the first target main node to acquire tasks matched with the request task number and the task type for feedback. According to the technical scheme of the embodiment of the invention, the working node actively requests the task type and the task quantity from the main node, so that the calculation amount ofthe main node is reduced, and the task concurrency supported by the system is improved.

Description

technical field [0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a distributed task scheduling method and system. Background technique [0002] At present, when users develop applications, they need to run applications in parallel on server clusters through a task scheduling system to improve application development. [0003] In the prior art, the commonly used task scheduling system is Yarn with a master-slave structure. However, Yarn is relatively difficult to use and is not friendly enough for users with poor development skills. Moreover, for applications with relatively simple business code logic, the development workload is relatively large, and the development efficiency is relatively low. At the same time, because Yarn needs to apply for resources first when starting tasks, the calculation workload of the master node is relatively heavy, which not only cannot meet the high requirements. Concurrent requir...

AI Technical Summary

Problems solved by technology

[0003] In the existing technology, the commonly used task scheduling system is Yarn with a master-slave structure. However, Yarn is relatively difficult to use and is not friendly enough for users with poor development skills.

Moreover, for applications with relatively simple business code logic, the development workload is relatively large, and the development efficiency is relatively low. At the same time, because Yarn needs to apply for resources first when starting tasks, the calculation workload of the master node is relatively heavy, which not only cannot meet the high requirements. Concurrent requirements may also cause system downtime due to failure of the master node

Images