Distributed dispatching system and method

Abstract

The invention provides a distributed dispatching system and method. The distributed dispatching system comprises a dispatching portion, an operation portion and a storage portion, wherein the dispatching portion comprises at least one dispatching unit, the operation portion comprises multiple operation units, the storage portion comprises at least one storage unit, and the dispatching units are used for extracting to-be-disposed tasks from a file system and dividing the to-be-disposed tasks into multiple subtasks, are also used for dispensing the multiple subtasks to the corresponding operation units and executing the corresponding subtasks by the operation units and are further used for recording record information in the subtask execution process and storing the record information in thestorage units. The single-point problem caused by the fact that a central device performs centralized task processing is solved, and when tasks fail or an execution device breaks down, other device nodes continue to perform execution, automatic multi-machine retrying of failed tasks is achieved, and timely and correct task operation is ensured. When other problems are caused in the task executionprocess, and an alarm notification can be also provided for a user through the system.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a distributed scheduling system and method. Background technique [0002] Heimdall is a massive data mining and analysis system with completely independent intellectual property rights, which can realize the mining and processing of massive data, and provide convenient and easy-to-use tools for data mining personnel and operational analysts to use. At present, when analysts use this system to query files, what they find is that the files are usually original logs, so the original logs need to be processed, processed, and analyzed again, which will undoubtedly increase the workload of analysts and is not conducive to improving analysis. The work efficiency of personnel, in order to provide convenience for analysts etc. at this time, it is necessary to directly realize further extraction and refinement of original logs in the Heimdall system. [0003] However, at present, when u...

AI Technical Summary

Problems solved by technology

In this method, all scheduling tasks are completed by the central equipment, and the scheduling information of data processing tasks is centralized and summarized on the management node of the central equipment, resulting in congestion of information flow. If the management node fails, it will affect the data processing of the entire system tasks, in addition, the task processing efficiency of the current system is low and cannot meet user needs

Images