An agent-based multi-environment application parallel cloud platform and its working method
A working method and cloud platform technology, applied in the field of cloud computing, can solve the problems of destroying Yarn's unified management and unified timing principles, resource leakage, and resource utilization can not be effectively improved, so as to improve cluster utilization and optimize resource management , save the effect of maintenance work
Active Publication Date: 2018-04-03
NANJING UNIV OF POSTS & TELECOMM
View PDF4 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0005] However, there are many contradictions in resource management between the two, which leads to the inability to effectively improve resource utilization.
For example, some programs running in Docker containers are separated from Yarn's management, which destroys Yarn's unified management and unified timing principles, resulting in resource leakage.
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0041] Below in conjunction with accompanying drawing, technical scheme of the present invention is described in further detail:
[0042] like figure 1 As shown, the present invention discloses a multi-environment application platform based on Yarn architecture, which builds a DAG computing framework based on Docker and Yarn, including two parts: DYbox and Docker.
[0043] The Agent is a module that integrates the DYbox and the Docker resource manager to realize efficient utilization of resources;
[0044] The DYbox is a program running module, which includes DYbox Client, ResourceManager, DYboxMaster, DYbox Driver, NodeManager, Container, DYbox Proxy, HDFS and State Server;
[0045] Described DYbox Client is DYbox client, and it is used for receiving user's task, and user's task is transformed into DAG task and it is subdivided into each sub-DAG task, and submits and stops sub-DAG task to ResourceManager;
[0046] The ResourceManager is a resource manager, which is used to ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses an Agent-based multi-environment application parallel cloud platform and a working method thereof, belonging to the field of cloud computing. The invention aims at the limited problem of the Yarn operating environment, establishes a multi-environment application platform composed of Yarn and Docker, and realizes the goal that the system can carry out tasks in different environments at the same time. At the same time, considering the difference between the resource management of Yarn and Docker at present, the traditional method is to directly bundle the two, destroying the original resource system and the unified timing principle, so this invention adds DYbox Proxy, which can manage the resources of both The system is integrated to avoid resource leakage, thereby achieving the purpose of greatly reducing the operation and maintenance costs in different environments.
Description
technical field [0001] The invention relates to the field of cloud computing, in particular to an Agent-based multi-environment application parallel cloud platform and a working method thereof. Background technique [0002] With the development of big data, a large amount of unstructured data and semi-structured data is growing exponentially, and it will take too much time and money to download it to a relational database for analysis. As a standard big data resource management system, MRv1 is widely used in various big data environments, but when the scale of the cluster continues to expand and the number of nodes exceeds 4000, it will show a certain degree of unpredictability. One of the biggest problems is cascading failures, where one failure can severely degrade the entire cluster by flooding the network due to attempts to replicate data and overload active nodes. Most importantly, it cannot support multi-tenancy, and cannot schedule processes for multiple users at the...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/08G06F9/455
CPCH04L67/10H04L67/1097G06F9/45558H04L67/60
Inventor 岳东杨雪健李诗玥邓松
Owner NANJING UNIV OF POSTS & TELECOMM