Distributed task scheduling system and method

Abstract

The embodiment of the invention provides a distributed task scheduling system and method. According to the scheme, a resource management module obtains cluster information of a multi-source big data computing environment and creates a resource pool corresponding to a resource manager queue of the big data computing environment; the model management module obtains a code package of a to-be-executed task; the task management module generates a to-be-executed task of a preset task type based on the code package, applies for a target resource in a target resource pool from the resource management module, and schedules the to-be-executed task to a task execution engine after resource application succeeds; and the task execution engine sends the to-be-executed task to a resource manager queue of the big data computing environment corresponding to the target resource pool. By applying the technical scheme provided by the embodiment of the invention, unified resource management of the multi-source big data computing environment is realized, management and scheduling of various types of tasks are realized, the service operation and maintenance cost in the big data computing environment is reduced, and the service code package management and online complexity is reduced.

Description

technical field [0001] The present application relates to the technical field of task scheduling, in particular to a distributed task scheduling system and method. Background technique [0002] With the rapid development of big data technology, application scenarios of machine learning, deep learning and reasoning based on big data emerge in endlessly. In the big data computing environment, first, the data needs to be preprocessed; then, the preprocessed data is input into the algorithm model for training, so as to generate the algorithm prediction model; after that, the algorithm prediction model is used for real-time data prediction. In the big data computing environment, the same preprocessing method can be used for different data, which leads to the problem of repeated development of the same processing or computing process. In order to solve this problem, in related technologies, common components are formed by extracting preprocessing operations or operators, that is,...

AI Technical Summary

Problems solved by technology

[0004] With business growth, changes in requirements, version updates and other driving factors, the requirements for the technical capabilities of the operation and maintenance personnel at each site are getting higher and higher, which makes the operation and maintenance costs in the big data computing environment higher, and the business The complexity of management is high, and the complexity of business algorithm package management and online is high

Images