Method for storing big data

Abstract

The embodiment of the invention provides a method for storing big data, and aims to realize centerless and efficient big data storage and solve the problem of single-point fault of the conventional Hadoop. The method provided by the embodiment of the invention is applied to two or more server clusters, and comprises the following steps: selecting at least two servers, starting GlusterFS service, sharing the local storage resources of the at least two servers into basic storage units of the GlusterFS, and constructing the basic storing units into a volume of the GlusterFS; selecting any server from the clusters for mounting the volume; starting the name node service of the Hadoop on the server on which the volume is mounted, and storing data of a name node on the mounted volume.

Description

technical field [0001] The invention relates to the field of computer data storage, in particular to a large data storage method. technical background [0002] Hadoop is a distributed computing platform that can be easily set up and used by users. Hadoop Distributed File System (HDFS) is designed as a distributed file system suitable for running on general-purpose hardware, with high efficiency, high scalability, and high fault tolerance. , Portability and other advantages. Hadoop is the best choice for setting up large-scale data centers, but there is a very dangerous flaw in Hadoop's architecture-single point of failure. Hadoop has a unique name node, which is responsible for managing the file system name space and controlling the access of external clients. Once the name node fails, it will largely lead to the loss of data in the cluster, resulting in catastrophic consequences. [0003] The existing technical solution is to back up the data of the name node on the netwo...

AI Technical Summary

Problems solved by technology

Hadoop is the best choice for setting up large data centers, but there is a very dangerous flaw in Hadoop's architecture - single point of failure

There is a unique name node in Hadoop, which is responsible for managing the file system namespace and controlling the access of external clients. Once the name node fails, it will largely lead to the loss of data in the cluster and have catastrophic consequences.

[0003] The existing technical solution is to back up the data of the name node on the network storage, such as the network file system NFS. Once a failure occurs, it is manually restored. During the restoration period, the data cannot be accessed and used, and the installation and maintenance of the network server is increased. cost

Images