A distributed data scheduling system and method

A distributed data and scheduling method technology, applied in the computer field, can solve problems such as information flow congestion, low efficiency of data processing tasks, and inability to satisfy extraction and refinement

Inactive Publication Date: 2018-12-18
BEIJING QIHOO TECH CO LTD
View PDF7 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In this method, all data scheduling tasks are uniformly completed by the central device, and the data processing task scheduling information is centralized and summarized on the management node of the central device, resulting in congestion of information flow...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A distributed data scheduling system and method
  • A distributed data scheduling system and method
  • A distributed data scheduling system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0076] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0077] In order to solve the above technical problems, an embodiment of the present invention provides a distributed data scheduling system. figure 1 A schematic structural diagram of a distributed data scheduling system according to an embodiment of the present invention is shown. see figure 1 , the distributed data scheduling system of this embodiment includes at least one scheduling component 10, multiple data mining units 20, ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a distributed data dispatching system and method. The system comprises at least one dispatching component. The dispatching component is adapted to obtain an off-line log to be processed from a file system and divide the off-line log to be processed into a plurality of sub-logs. The scheduling component is further adapted to distribute the plurality of sub-logs to a pluralityof data mining units, and the data mining unit mines log metadata from the sub-logs according to preset rules; the scheduling component is further adapted to store the log metadata and other processinformation into a storage component including a plurality of preset databases. The single point problem caused by the centralized processing of tasks by the central equipment is solved. When the taskfails or the executing equipment fails, the other equipment nodes continue to execute the task, thereby realizing the automatic multi-machine retry of the task failure, ensuring the timely and correct operation of the task. The system can also alert the user when other problems occur during the execution of the task.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a distributed data scheduling system and method. Background technique [0002] Heimdall is a massive data mining and analysis system with completely independent intellectual property rights, which can realize the mining and processing of massive data, and provide convenient and easy-to-use tools for data mining personnel and operational analysts to use. At present, when analysts use this system to query files, what they find is that the files are usually original logs, so the original logs need to be processed, processed, and analyzed again, which will undoubtedly increase the workload of analysts and is not conducive to improving analysis. The work efficiency of personnel, in order to provide convenience for analysts etc. at this time, it is necessary to directly realize further extraction and refinement of original logs in the Heimdall system. [0003] However, at present, w...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
Inventor 王肖磊王志超李敬轩
Owner BEIJING QIHOO TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap