Job scheduling management system and method

Abstract

The invention provides a job scheduling management system and method. The job scheduling management system structurally comprises an FTP (File Transfer Protocol) server, a client, a control node and a plurality of processing nodes. The job scheduling management method includes that a task node sends task running state information including middle progress state, error or exception of a task to the control node, and the control node executes exception processing or stopping according to an exception or error condition; the control node starts a heartbeat test to test whether a node is alive or not when the control node do not receive heartbeat information of the task node, and the control node reschedules all uncompleted tasks when the node crashes. Compared with the prior art, the job scheduling management system and method improves the level stack of large data processing software, enables Hadoop to break resources telescopic bottlenecks and business expansion limit of a single data center, promotes the integration of multiple data centers, and further expands computing capabilities and improves the resource utilization rate.

Description

technical field [0001] The invention relates to the technical field of cloud computing, in particular to a multi-data center and cross-Hadoop cluster job scheduling management system and method. Background technique [0002] In recent years, with the in-depth development of informatization construction, a large number of front-end devices, such as sensors, video and mobile terminals, have been widely used, generating massive data, such as access records, business video and audio, pictures and other semi-structured and unstructured The skyrocketing data has made the current storage and computing architecture unable to meet the development needs of "big data". As a strategic resource, the importance of data is unquestionable. On the basis of data integration and storage, how to quickly analyze and mine valuable information from massive data to improve the analysis, decision-making and command levels of government or industry departments , has become a hot issue in the field o...

AI Technical Summary

Problems solved by technology

However, the current various open source versions and various hairstyle versions of Hadoop cannot meet the job scheduling between Hadoop clusters across data centers, and the main problems are:

[0007] The use of message middleware can meet the requirements of secure authentication access and asynchronous and reliable transmission, and establish a loosely coupled two-level Hadoop cluster architecture. However, the existing message middleware lacks the monitoring method for access node status and running jobs, and only supports Establish a static broadcast topic, lack a dynamic multicast mechanism, and cannot realize the runtime customization requirements of routing groups

In addition, the existing message middleware needs to maintain the message state in memory, or use the database to persist the message, and its transmission protocol requires a lot of fragmentation and assembly, serialization and deserialization overhead, when the file size is too large , for example, when it exceeds 1 GB, it will cause a serious performance bottleneck. However, in Hadoop-based big data scenarios, large files are quite common. Therefore, how to achieve high-speed transmission of large files is the key to cross-data center Hadoop cluster job scheduling management

Images