Method for fast constructing Hadoop cluster based on container cloud technology

Abstract

The invention discloses a method for fast constructing Hadoop cluster based on container cloud technology. The method comprises the following steps: (1) determining selected physical server number according to actual demand, regarding the entire selected physical server as a cluster, and assigning a master node of the cluster, and distributing actual deployment nodes, namely as the slave node; (2) installing a CentOS7 operation system for each physical server in the cluster; (3) installing and starting a container orchestration service Kubernetes and the network service Flannel on each master node of the cluster by use of a yuminstall command; (4) installing MySQL database, a mirror image warehouse service and management portal; (5) logging a management portal interface by use of an administrator account, establishing a user; and logging the management user by use of the established user, creating the cluster environment of the user, and constructing a Hadoop cluster.

Description

technical field [0001] The invention relates to the technical field of creating Hadoop clusters, in particular to a method for rapidly building Hadoop clusters based on container cloud technology. Background technique [0002] At present, Hadoop clusters are usually built based on virtual machines or PCs; after setting the scale of the cluster in advance (including the number of master nodes and slave nodes, etc.), it is necessary to establish a specified number of virtual machines according to the preset scale or purchase a specified Number of PCs; after these servers are established, you need to install the operating system one by one, and install the JDK and other services necessary to build the Hadoop cluster, specify the host name of each server, and configure the SSH password-free between each two servers Authentication; finally, you need to deploy Hadoop on the master node, modify the configuration, and distribute it to each of the remaining servers in the cluster; af...

AI Technical Summary

Problems solved by technology

[0004] However, due to the large number of vendors and numerous release versions, OpenStack has various technical difficulties; first, OpenStack is far from enough in terms of integration, scalability, and stability; Nova, Swift, Cinder, and Neutron use their own database storage Configuration information, installation and upgrades are time-consuming and laborious;

[0005] Secondly, OpenStack lacks integrity; OpenStack can only provide three cloud products: computing, storage, and network. If you build a Hadoop cluster based on OpenStack, you need to get through the account, security, management operation and maintenance, and monitoring systems between OpenStack and Hadoop. The process is complicated and cumbersome;

[0006] Furthermore, OpenStack cannot provide end-to-end service guarantee; users only get scattered "framework" and components, and must manually integrate the functions of multiple vendors and multiple versions;

[0007] Finally, OpenStack lacks a common basic version; there are currently more than 20 customized versions of OpenStack that can be downloaded from manufacturers, and customers do not know which version to choose, let alone how to combine, mix and match and migrate between different versions according to their own needs ;The above defects lead to the invention patent disclosed by Jiangsu Internet of Things Research and Development Center. It takes a long time to install and implement the system, the coupling is also high, and the system stability cannot be guaranteed. optimize

[0008] Using multiple PCs or physical servers to build a Hadoop cluster has relatively high cost and low resource utilization; using virtual machines to provide the entire virtualization hardware layer, but the cost of use will also increase significantly, and resource utilization is also low; In addition, After using a PC or a virtual machine to build a Hadoop cluster, if the cluster environment is damaged during use, it is difficult to find the problem immediately; after the environment is found to be damaged, it is necessary to rebuild the environment of the damaged single or multiple servers or virtual machines, and deal with the problem. timeliness cannot be guaranteed

Images