Big data task scheduling system

Abstract

The invention relates to a big data task scheduling system, and belongs to the technical field of computers. The system comprises a user interface UI which is used for generating a task scheduling request by adopting interface operation by a user; the management node is used for receiving a task scheduling request sent by a user interface (UI) and selecting a working node according with a selection standard to distribute tasks according to the task scheduling request; the distributed message middleware is used for temporarily storing the tasks allocated by the management node; the working node is used for executing the task allocated by the management node; and the etcd database is deployed in the system in a containerization manner and is used for recording registration and synchronization information of the management nodes and the working nodes so as to realize the function of a distributed lock. According to the system, resource elastic capacity expansion can be achieved, the concurrency degree is improved, and the recovery time is shortened when the nodes fail.

Description

technical field [0001] The invention belongs to the technical field of computers, and in particular relates to a big data task scheduling system. Background technique [0002] The development languages ​​supported by the technical framework of the big data platform are diverse, and the backgrounds of developers are also very different, which results in many different types of programs (tasks) running on the big data platform, such as: MapReduce, Hive, Spark, Shell, Python, etc. And there is often a certain dependency between these tasks, and it is obviously inefficient to perform tasks manually at this time. [0003] The emergence of the big data task scheduling system frees developers from needing to pay attention to how tasks are submitted, scheduled, and executed, whether resource allocation is reasonable, whether dependencies are satisfied, etc., so that developers can focus more on the business instead of Concerned about when the data is output, data quality issues, e...

AI Technical Summary

Problems solved by technology

[0015] 1) Decentralization problem: Since there is no "manager" node, each node needs to communicate with other nodes to obtain the necessary machine information, and the unreliability of distributed system communication greatly increases the above functions difficulty of implementation

If the new scheduling process has a large amount of data or complex business logic, it may have a significant negative impact on other processes or even the entire system, resulting in process interruption, business processing blockage, system crashes and other consequences

[0017] 2) High availability issues:

[0018] At present, it is difficult to really achieve high availability of management nodes. It is only through zookeeper to ensure that after one machine goes down, another machine is reinitialized.

In a production environment, the management node needs to initialize a large amount of data in memory, which takes a long time

[0019] Moreover, the database that stores task flow metadata in the cluster needs to be manually configured for high availability. This kind of high availability based on the database itself still has the possibility of single point of failure and complex configuration

Images