Automatic deployment system and method of hadoop cluster

A Hadoop cluster and automatic deployment technology, applied in the Hadoop field, can solve problems such as complex framework, immature management, and increased labor costs, and achieve the effect of simple framework structure and low maintenance cost

Active Publication Date: 2013-04-24
SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
View PDF4 Cites 45 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

At the same time, the Mesos system has the following disadvantages: the framework is too complex. In order to support other computing frameworks, not only the source code of Mesos must be very familiar, but also other computing frameworks must be very proficient, which obviously increases the labor cost; Mesos implements hadoop-related interfaces through SWIG and encapsulates them into Java objects
In short,

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Automatic deployment system and method of hadoop cluster
  • Automatic deployment system and method of hadoop cluster
  • Automatic deployment system and method of hadoop cluster

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0068] Example

[0069] It should be noted that the hadoop cluster automatic deployment system provided in this embodiment manages the hadoop system through a two-tier structure of jobs and tasks. A job is an execution unit of the system and consists of tasks. A task is an execution process of an application.

[0070] We define the deployment process of Hadoop cluster as a job, which includes three types of tasks. The first type of task deploys the master, and you can specify the master to any machine; the second type of task deploys multiple slaves, which are allocated arbitrarily by the system, but the machine to be deployed cannot be specified separately, that is, slaves should be deployed on different machines and cannot be deployed On machines that have already deployed slaves, the machines deployed by slaves are also assigned by the system. Generally speaking, no specific machines are specified. The basic flow for both types of tasks is the same. They are included in...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides an automatic deployment system and method of a hadoop cluster. The system comprises a command controller, a master controller, a deployer, a storing structure, an environment configurator, a resource distributor, a parameter configurator, a data synchronizer, a starter, a data downloader, a monitor and a cleaner, the command controller is used for generating operation description information according to user demands and submitting operations, the master controller is used for receiving and processing the operation description information submitted by the command controller, the deployer is used for providing heartbeat information of the master controller, receiving tasks distributed by the master controller and deploying relevant tasks, the storing structure is used for storing application data, computed application results and logs which are submitted by a user, the environment configurator, the resource distributor, the parameter configurator, the data synchronizer, the starter, the data downloader, the monitor and the cleaner are further arranged through deployment modules in the deployer so that automatic deployment and recycling of the hadoop cluster can be achieved, and the operation of a program can be dynamically monitored.

Description

technical field [0001] The invention relates to the hadoop field, in particular to an automatic deployment system and method for hadoop clusters. Background technique [0002] Hadoop is a distributed system infrastructure. Users can develop distributed programs without knowing the underlying details of the distribution, and make full use of the power of the cluster for high-speed computing and storage. Hadoop implements a distributed file system (Hadoop Distributed FileSystem), referred to as HDFS. HDFS is highly fault-tolerant and designed to be deployed on inexpensive hardware. [0003] Currently, Hadoop clusters are configured manually, which is cumbersome, inefficient, and error-prone. Especially in a large-scale data center environment, elastic management such as dynamic expansion and contraction of the system cannot be performed. Therefore, realizing hadoop automatic configuration and automatic deployment is an urgent problem to be solved. At present, a typical sys...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/50
Inventor 须成忠王明阳洪爵皇甫亚光
Owner SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products