Automatic deployment system and method of hadoop cluster

Abstract

The invention provides an automatic deployment system and method of a hadoop cluster. The system comprises a command controller, a master controller, a deployer, a storing structure, an environment configurator, a resource distributor, a parameter configurator, a data synchronizer, a starter, a data downloader, a monitor and a cleaner, the command controller is used for generating operation description information according to user demands and submitting operations, the master controller is used for receiving and processing the operation description information submitted by the command controller, the deployer is used for providing heartbeat information of the master controller, receiving tasks distributed by the master controller and deploying relevant tasks, the storing structure is used for storing application data, computed application results and logs which are submitted by a user, the environment configurator, the resource distributor, the parameter configurator, the data synchronizer, the starter, the data downloader, the monitor and the cleaner are further arranged through deployment modules in the deployer so that automatic deployment and recycling of the hadoop cluster can be achieved, and the operation of a program can be dynamically monitored.

Description

technical field [0001] The invention relates to the hadoop field, in particular to an automatic deployment system and method for hadoop clusters. Background technique [0002] Hadoop is a distributed system infrastructure. Users can develop distributed programs without knowing the underlying details of the distribution, and make full use of the power of the cluster for high-speed computing and storage. Hadoop implements a distributed file system (Hadoop Distributed FileSystem), referred to as HDFS. HDFS is highly fault-tolerant and designed to be deployed on inexpensive hardware. [0003] Currently, Hadoop clusters are configured manually, which is cumbersome, inefficient, and error-prone. Especially in a large-scale data center environment, elastic management such as dynamic expansion and contraction of the system cannot be performed. Therefore, realizing hadoop automatic configuration and automatic deployment is an urgent problem to be solved. At present, a typical sys...

AI Technical Summary

Problems solved by technology

At the same time, the Mesos system has the following disadvantages: the framework is too complex. In order to support other computing frameworks, not only the source code of Mesos must be very familiar, but also other computing frameworks must be very proficient, which obviously increases the labor cost; Mesos implements hadoop-related interfaces through SWIG and encapsulates them into Java objects

In short, it is necessary to modify the source code of Hadoop, which increases the cost of development and maintenance; Mesos is not mature enough for resource management, and the slave node resource information is only the size of Mem and the number of CPUs, and the management is too rough; Mesos is not effective for programs The local monitoring mechanism cannot obtain the running status of the program in real time

Images