Unlock instant, AI-driven research and patent intelligence for your innovation.

Intelligent distributed calculation management system and method based on DAG (Directed Acyclic Graph)

A technology of distributed computing and management methods, applied in the field of distributed computing management, can solve problems such as inability to proceed, waste of system resources, unsuitable for small file storage, etc., to achieve flexible scheduling strategies, improvement of various indicators, and flexible scheduling strategies Effect

Inactive Publication Date: 2018-08-10
北京易沃特科技有限公司
View PDF5 Cites 13 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Distributed systems in the prior art (for example, Hadoop distributed systems) are all oriented to batch processing systems and are ideal technologies for processing massive amounts of data, but these technologies are not suitable for serial operations
For example, MapReduce is based on GFS (Google File System) or HDFS (Hadoop File System). One of the characteristics of this kind of master-slave distributed file system is that it is not suitable for the storage of small files. The entire system causes a huge waste of resources
[0006] 2) only have a fixed parallel granularity
Distributed systems in the prior art only have a fixed parallel granularity, thus limiting the range of jobs they can accept. Usually, some jobs with more complex logic are difficult to deploy on these systems. For example, MapReduce is difficult to handle machine learning iterative algorithm
[0007] 3) The basic unit of scheduling is large
Larger scheduling units will bring many resource allocation problems. For example, when a job only occupies IO resources in the last part, but it cannot release other resources such as CPU, GPU, etc., other jobs cannot be performed, resulting in waste of resources

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Intelligent distributed calculation management system and method based on DAG (Directed Acyclic Graph)
  • Intelligent distributed calculation management system and method based on DAG (Directed Acyclic Graph)
  • Intelligent distributed calculation management system and method based on DAG (Directed Acyclic Graph)

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0052] combine figure 2 , the DAG-based intelligent distributed computing management system provided in this embodiment, the system has a master-slave distribution structure, including: a root node, at least one core node that is connected to the root node, and a core node that is connected to each core node operating node;

[0053] The operation node is a leaf node or a core node, and the leaf node is the basic task execution unit in the system;

[0054] Wherein, each core node corresponds to a running pool, and the running pool has a DAG graph structure; each running pool has a parent running pool and / or a child running pool.

[0055]Further, the leaf nodes are functions, modules or files.

[0056] The DAG-based intelligent distributed computing management system provided by the embodiment of the present invention is a distributed software architecture. Based on this architecture, job tasks can be subdivided and divided into execution units with smaller granularity for ta...

Embodiment 2

[0062] combine image 3 The DAG-based intelligent distributed computing management method provided in the embodiment of the present invention is applied to the system provided in Embodiment 1, and the method includes:

[0063] Step S1, obtaining job tasks received by the root node;

[0064] Step S2, splitting the job tasks into job DAG graphs;

[0065] Step S3, adding the job DAG graph into the historical running pool, and merging according to the set rules to form the current running pool;

[0066] In step S4, task allocation is performed according to the vertices of the current running pool, so as to implement granular processing of job tasks.

[0067] The DAG-based intelligent distributed computing management method provided by the embodiment of the present invention is applied to the system provided in the first embodiment, wherein the system is a distributed software architecture, based on this architecture, it is possible to subdivide job tasks and disassemble Divide ...

Embodiment 3

[0100] This embodiment is a specific exemplary description, and specifically, the system in Embodiment 1 or Embodiment 2 is used to establish a machine learning platform. The established platform needs to meet the following functions: support multi-users, be able to support various databases, and keep the overall utilization rate of the system at a high level.

[0101] A complete machine learning task usually requires data integration, data preprocessing, training, evaluation and comparison. The task of data integration is to collect the data that needs to be trained from different storage media, so this process is usually mainly transmission and retrieval operations; data preprocessing usually includes data cleaning, missing value filling, etc.; training It is a work that requires iterative calculations, and is usually the most time-consuming in the entire machine learning process.

[0102] According to the process of machine learning, we divide this platform into three kern...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides an intelligent distributed calculation management system and method based on a DAG (Directed Acyclic Graph). The method comprises the following steps that: obtaining a job taskreceived by a root node; splitting the job task into a job DAG graph; adding the job DAG graph into a historical operation pool, and combining according to a set rule to form a current operation pool;and according to the vertex of the current operation pool, carrying out task distribution to carry out sub-granularity processing on the job task. By use of the intelligent distributed calculation management system and method based on the DAG, the system is of distributed software architecture; on the basis of the architecture, the job tasks can be subdivided and are split into execution units with smaller granularities for task execution, so that a scheduling strategy can be more flexible, the load of the system is balanced, and fault tolerance and handling capacity indexes are improved.

Description

technical field [0001] The invention relates to the field of distributed computing management, in particular to a DAG-based intelligent distributed computing management system and method. Background technique [0002] With the rise of big data and deep learning, massive data and calculations will generate various long-time jobs. How to efficiently and stably handle long-time jobs has become a very important issue. For long-running jobs, stability and high throughput are far more important than low latency, which also determines that such jobs are not suitable for running in systems that pursue real-time performance. Parallel computing is an important technical idea to accelerate the solution of problems. The dedicated GPU and TPU developed for deep learning, and the clusters using open source distributed frameworks such as Hadoop and Spark are important practices of parallel computing ideas. [0003] In the prior art, for example, the core of Hadoop distributed system compu...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50G06F9/38
CPCG06F9/3885G06F9/5083G06F2209/5011
Inventor 胡泽琛李守东涂龙马迪
Owner 北京易沃特科技有限公司