Unlock instant, AI-driven research and patent intelligence for your innovation.

HDFS cluster based storage node transfer method and device

A technology of storage node and transfer method, applied in the field of transfer of storage nodes, can solve the problems of reducing the availability of HDFS cluster, increasing the overall work pressure of the cluster, reducing the work efficiency of HDFS cluster, etc., so as to achieve the effect of ensuring availability and reducing work pressure.

Inactive Publication Date: 2018-11-06
ZHENGZHOU YUNHAI INFORMATION TECH CO LTD
View PDF4 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Due to the working nature of HDFS, the data in the faulty node in the cluster will be transferred to the normal nodes in the cluster. Therefore, when the number of nodes configured with HDFS in the cluster is small, or the nodes in the HDFS cluster have a large-scale failure When the faulty node transfers data to other normal nodes, it will increase the overall working pressure of the cluster and reduce the working efficiency of the HDFS cluster, which in turn will reduce the availability of the HDFS cluster

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • HDFS cluster based storage node transfer method and device
  • HDFS cluster based storage node transfer method and device
  • HDFS cluster based storage node transfer method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0041] figure 1 It is a flow chart of an HDFS cluster-based storage node transfer method provided by an embodiment of the present invention. Please refer to figure 1 , the specific steps of the method for transferring storage nodes based on HDFS clusters include:

[0042] Step S10: Obtain the running status of the storage nodes in the cluster through the monitoring node.

[0043] In this step, the state of establishing a connection between the monitoring node and the storage node in the cluster is for monitoring the operation status of the storage node and recording the operation status. It should be noted that the topology of the monitoring nodes can be specifically connected to and controlled by a master monitoring node to form a monitoring node cluster. The master monitoring node is used to record the monitoring data of the slave monitoring nodes, and control the slave monitoring nodes to perform corresponding operations according to the monitoring situation of the slave...

Embodiment 2

[0051] figure 2 It is a flow chart of another HDFS cluster-based storage node transfer method provided by an embodiment of the present invention. figure 2 In steps S10-S12 and figure 1 Same, no more details here.

[0052] Such as figure 2 As shown, as a preferred embodiment, the method further includes:

[0053] Step S20: adding a new non-storage node, configuring and recording address information of the new non-storage node.

[0054] It can be understood that adding a non-storage node in the cluster is equivalent to adding a backup storage node, thereby avoiding the situation that no non-storage node can be configured as a storage node when a storage node fails. The purpose of configuring and recording the address information of the new non-storage node is to find the corresponding non-storage node through the address information in the subsequent steps. This step can be performed before any step or simultaneously with any step, and is not specifically limited here.

...

Embodiment 3

[0070] The embodiment of the method for transferring storage nodes based on HDFS clusters has been described in detail above, and the present invention also provides a device for transferring storage nodes based on HDFS clusters corresponding to the method. The embodiments of the method part correspond to each other, so for the embodiments of the device part, please refer to the description of the embodiments of the method part, and details will not be repeated here.

[0071] image 3 A structural diagram of a transfer device for storage nodes based on HDFS clusters provided by an embodiment of the present invention. The transfer device based on the storage node of the HDFS cluster provided by the embodiment of the present invention specifically includes:

[0072] The monitoring module 10 is configured to obtain the running status of the storage nodes in the cluster through the monitoring node.

[0073] Judging module 11, configured to judge whether the storage node fails ac...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses an HDFS cluster based storage node transfer method and device. The method comprises the steps of obtaining a running state of a storage node in a cluster through a monitoring node; judging whether the storage node has a fault based on the running state; if so, obtaining a non-storage node in the cluster and performing related configuration of the HDFS on the non-storage node to enable the non-storage node to be as a new storage node, wherein the monitoring node, the storage node and the non-storage node enable yarn and Hbase. Visibly, when the storage nodes in the cluster are few or most of the storage nodes in the cluster have faults, the new storage node can be supplemented in the cluster. The number of available nodes in the HDFS cluster is guaranteed to be not changed a lot, the working pressure of each node in the HDFS cluster is lightened and the integral working efficiency and the integral availability of the HDFS cluster are further guaranteed. In addition, the HDFS cluster based storage node transfer device also has the effects abovementioned.

Description

technical field [0001] The invention relates to the field of HDFS distributed file systems, in particular to a method and device for transferring storage nodes based on HDFS clusters. Background technique [0002] Hadoop is a distributed system infrastructure developed by the Apache Foundation. Users can develop distributed programs without knowing the underlying details of the distribution, and then can make full use of the integrity of the cluster for high-speed computing and storage. The architecture of Hadoop mainly realizes the distributed storage of data through the HDFS distributed file system, which can provide efficient data support for a series of data processing frameworks of Hadoop. [0003] Since the HDFS distributed file system has cluster characteristics, when a node in the HDFS cluster fails, the standby node can be enabled or the data of the failed node can be transferred to other nodes in the HDFS cluster. Nodes or nodes for data transfer must be configur...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): H04L12/24H04L29/08G06F17/30
CPCH04L41/0668H04L41/0816H04L67/1097
Inventor 白培翔秦楠楠尹艳艳
Owner ZHENGZHOU YUNHAI INFORMATION TECH CO LTD