Automatic slave node service recovering method of hadoop system

Abstract

The invention provides an automatic slave node service recovering method of a hadoop system. The automatic slave node service recovering method of the hadoop system comprises the following steps that a Gluster distributed file storage system is deployed in the middle of a node cluster, and a storage space is provided; the storage space is mounted on a main-controlling node, the main-controlling node is configured, and the establishment of the hadoop system is finished. The configuration comprises that the data of the main-controlling node is stored in the storage space. The slave node recovering service comprises the steps that the storage space in which the main-controlling node is stored is mounted to a local slave node, whether namespaceID of the local slave node is the same with the namespaceID of the storage space is checked, if the namespacceID of the local slave node is different with the namespaceID of the storage space, the namespaceID of the local slave node is revised as the namespaceID of the storage space, and the slave node starts the service. According to the automatic slave node service recovering method of the hadoop system, automatic recovering of service of the slave node after faults is accomplished, the work of manual searching and replacement is avoided, the risk of the mistaken manual operation is reduced, and investment of human cost is reduced.

Description

technical field [0001] The invention relates to the field of computer distributed computing, in particular to a method for automatically recovering slave node services in a Hadoop system. Background technique [0002] Hadoop is a distributed computing platform that can be easily set up and used by users. Hadoop Distributed File System (HDFS) is designed as a distributed file system suitable for running on general-purpose hardware, with high efficiency, high scalability, and high fault tolerance. , Portability and other advantages. Therefore hadoop is the best choice for setting up large data centers. Hadoop has a unique master control node (also known as namenode) and multiple slave nodes. The master control node is responsible for managing the file system namespace and controlling the access of external clients, and assigning tasks to the slave nodes for execution. The erection and startup process of the traditional Hadoop distributed cluster: 1. First, start a node as th...

AI Technical Summary

Problems solved by technology

In practical applications, the server of the slave node is restarted due to abnormal conditions such as power outages. After restarting, the stored namespace ID and the namespace ID of the master node often appear inconsistent, resulting in the node not being able to join the hadoop system, thus failing to restore the slave node The service is often restored by the administrator to find all abnormal nodes and manually change the namespaceID of the slave node to be consistent with the master node. Since Hadoop application scenarios are mostly large-scale data centers, troubleshooting and modification are heavy tasks , and introduces the risk of human error

Images