A task scheduling processing method, device and system

Abstract

The present application provides a task scheduling processing method, device and system. The method may include mapping the task processing channel information in the resource pool to the channel resource information of the configuration center; when receiving at least one job task scheduling request, querying the configuration center whether there is any idle time for processing the job task channel resource; when the query result of the configuration center is that there is the idle channel resource, select the task processing channel corresponding to the scheduling request from the resource pool according to the resource allocation priority set for the job task assigned to the corresponding job tasks. Using various embodiments in this application, the processing method of configuring multiple channels can be used to solve the channel blocking problem caused by the deadlock of a single job, and it can provide parallel submission and parallel processing of multiple job tasks, and quickly and efficiently allocate the required tasks for the job tasks resources, greatly improving task scheduling processing efficiency and resource utilization.

Description

technical field [0001] The present application belongs to the field of computer data processing, and in particular relates to a task scheduling processing method, device and system. Background technique [0002] With the development of information technology and the improvement of data business processing requirements, distributed systems have received more and more attention because of their high cohesion and transparency. [0003] In the distributed task processing scenario, usually each task has a Driver (main scheduler or driver) to schedule multiple Excutors (executors or execution programs). figure 1 It is a schematic diagram of the task scheduling processing scheme in the existing distributed scenario, such as figure 1 As shown, a physical cluster can be deployed in the prior art, and the servers in the physical cluster usually adopt a master-slave architecture mode, that is, one master (master server) manages and schedules multiple slaves (slave servers). The runni...

AI Technical Summary

Problems solved by technology

This results in a single job hogging all resources, wasting other idle resources when one job requires less

Moreover, under the first-in-first-out mechanism, if the execution time of a job is very long, the subsequent job tasks will not be allocated resources, channel congestion will occur, and the job will starve to death.

Therefore, the single-channel task scheduling processing mechanism in the distributed scenario in the prior art still has the problems of waste of resources, failure to execute subsequent jobs when the job is abnormal, and low processing efficiency

Images