HDFS cluster high-availability deployment method, system and device and storage medium

Abstract

The embodiment of the invention provides an HDFS cluster high-availability deployment method, system and device and a storage medium. The method comprises the following steps: creating a management node mirror image and a data node mirror image for the HDFS cluster, wherein the management node mirror image and the data node mirror image respectively comprise a configuration file for configuring the HDFS cluster into a high-availability mode; respectively uploading the management node mirror image and the data node mirror image to a mirror image warehouse; deploying the HDFS cluster to a plurality of host machines through a Kubernetes platform, and enabling the Kubernetes platform to creats a main management container in a first host machine based on the management node mirror image uploaded to the mirror image warehouse, and create a hot standby management container in a second host machine, and creat a data storage container in at least one third host machine based on the data node mirror image uploaded to the mirror image warehouse. According to the embodiment of the invention, the HDFS cluster can be operated in cheap hardware equipment, so that the resource utilization rate ofthe HDFS cluster is improved, and the digital transformation threshold and capital investment of an enterprise are reduced.

Description

technical field [0001] Embodiments of the present invention relate to the field of big data applications, and more specifically, relate to a HDFS (Hadoop Distributed File System, Hadoop Distributed File System) cluster high-availability deployment method, system, device and storage medium. Background technique [0002] At present, big data is playing an increasingly obvious role in work and life. For example, big data assists shopping platforms to recommend products suitable for customers, big data assists in avoiding traffic jams, big data assists in health checkups, and big data in entertainment. Due to the huge amount of data, the requirements for calculation speed and accuracy are relatively high. Simply increasing the number of processors to enhance the computing power of a single computer has not achieved the expected effect. At present, the direction of big data processing is gradually developing towards distributed computing clusters. Computers distributed in differe...

AI Technical Summary

Problems solved by technology

[0012] The embodiments of the present invention aim at the waste of machine resources in the traditional big data high-availability deployment method based on physical machines or virtual machines, unable to provide load balancing in the cluster, creating and deploying application components with low efficiency and time-consuming, and cluster expansion with low efficiency and time-consuming , and problems that cannot be checked and automatically repaired, provide a high-availability deployment method, system, equipment and storage media for HDFS clusters

Images