Distributed system job scheduling system and method

A distributed system and job scheduling technology, applied in database distribution/replication, resource allocation, program startup/switching, etc., can solve the problem of inability to concurrency and resource inclination, lack of self-learning and self-feedback capability, and difficulty in data backtracking, etc. It can reduce the complexity and coupling of dependencies, realize the reasonable allocation of resources, and improve the scheduling efficiency.

Active Publication Date: 2017-10-10
INDUSTRIAL AND COMMERCIAL BANK OF CHINA
View PDF4 Cites 24 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] 2. The function is single, and only the simplest job scheduling function is realized, and the scheduling job cannot be reasonably analyzed, and the job execution path is reasonably configured.
[0005] 3. Lack of self-learning and self-feedback capabilities for historical operation experie...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed system job scheduling system and method
  • Distributed system job scheduling system and method
  • Distributed system job scheduling system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0033] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with the embodiments and accompanying drawings. Here, the exemplary embodiments and descriptions of the present invention are used to explain the present invention, but not to limit the present invention.

[0034] In order to improve the reading speed of massive jobs and the concurrency of analysis, the system in the embodiment of the present invention will be deployed in a distributed system, based on the sharing mechanism of the distributed file system, making full use of the computing power of the distributed cluster spark, overcoming the For the problem of slow traditional complex machine learning training speed, realize fast traversal, iterative and recursive analysis operations, layer-by-layer module formation of job grouping, correlation, dependency, serial parallel, mutual exclusion, loop, ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a distributed system job scheduling system and method. The system comprises a job extracting device for extracting multiple jobs to be scheduled in a distributed file system, a consanguinity analysis device for analyzing the multiple jobs to be scheduled to obtain the consanguinity among the multiple jobs to be scheduled, a complete job path construction device for constructing all job execution paths among the jobs to be scheduled according to the consanguinity, a critical path recognition device for recognizing a key job execution path from all job execution paths, a job grouping device for splitting and grouping all the job execution paths to obtain grouped job execution paths, and a job execution device for executing the jobs to be scheduled according to the key job execution path and the grouped job execution paths. According to the technical scheme, massive job scheduling sequences are automatically configured, resources are reasonably assigned, data is traceable, and the job scheduling efficiency is improved.

Description

technical field [0001] The invention relates to the technical field of job scheduling, in particular to a job scheduling system and method for a distributed system. Background technique [0002] In recent years, with the development of information technology and the Internet, the total amount of data in the world has reached 500 billion GB, and the growth rate is even higher than the speed of NASA rockets. People have quickly entered the era of information overload from the era of information scarcity. The main challenge in this era is: how to extract interesting information from a large amount of information, and then apply it to analysis and decision-making. From data collection, extraction, conversion, loading, data mart generation, to the formation of the data cube for decision-making, thousands of processing programs are involved, and the relationship between processing programs is intricate and intricate. It is not an exaggeration to describe it as a spider web of hom...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/48G06F9/50G06F17/30
CPCG06F9/4881G06F9/5083G06F16/27
Inventor 王备蔡良良俞斌江军华
Owner INDUSTRIAL AND COMMERCIAL BANK OF CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap