A distributed data scheduling system and method

Abstract

The invention provides a distributed data dispatching system and method. The system comprises at least one dispatching component. The dispatching component is adapted to obtain an off-line log to be processed from a file system and divide the off-line log to be processed into a plurality of sub-logs. The scheduling component is further adapted to distribute the plurality of sub-logs to a pluralityof data mining units, and the data mining unit mines log metadata from the sub-logs according to preset rules; the scheduling component is further adapted to store the log metadata and other processinformation into a storage component including a plurality of preset databases. The single point problem caused by the centralized processing of tasks by the central equipment is solved. When the taskfails or the executing equipment fails, the other equipment nodes continue to execute the task, thereby realizing the automatic multi-machine retry of the task failure, ensuring the timely and correct operation of the task. The system can also alert the user when other problems occur during the execution of the task.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a distributed data scheduling system and method. Background technique [0002] Heimdall is a massive data mining and analysis system with completely independent intellectual property rights, which can realize the mining and processing of massive data, and provide convenient and easy-to-use tools for data mining personnel and operational analysts to use. At present, when analysts use this system to query files, what they find is that the files are usually original logs, so the original logs need to be processed, processed, and analyzed again, which will undoubtedly increase the workload of analysts and is not conducive to improving analysis. The work efficiency of personnel, in order to provide convenience for analysts etc. at this time, it is necessary to directly realize further extraction and refinement of original logs in the Heimdall system. [0003] However, at present, w...

AI Technical Summary

Problems solved by technology

In this method, all data scheduling tasks are uniformly completed by the central device, and the data processing task scheduling information is centralized and summarized on the management node of the central device, resulting in congestion of information flow. If the management node fails, it will affect the data of the entire system Processing tasks, and the data processing tasks of the current system are inefficient and cannot meet the needs of further extraction and refinement of original logs directly in the system

Images