Heartbeat interval dynamic adjustment method, device and system based on node heat

Abstract

The invention provides a heartbeat interval dynamic adjustment method, device and system based on node heat. The method comprises the following steps: separately obtaining access frequencies of DataNode slave nodes i in the first m continuous periods of periods where a current moment j is located, wherein m is a preset integer parameter; calculating the predicted access heat of the DataNode slavenode i at the current moment j according to the access frequencies of the first m continuous periods; calculating a heartbeat sending interval of the DataNode slave node i at the current moment j according to the predicted access heat; and performing automatic adjustment processing according to the heartbeat sending interval. According to the method, the access heat of the DataNode slave nodes ineach period is calculated in order to predict the access heat of the next moment and calculate a heartbeat sending interval through the access heat; therefore, the heartbeat sending interval of the DataNode slave node is dynamically adjusted, the defect that in the prior art, the heartbeat sending interval cannot be dynamically adjusted according to the access frequency of the DataNode slave node,namely the access heat of the DataNode slave node, is overcome, and the performance of an HDFS system is improved.

Description

technical field [0001] The present invention relates to the field of big data technology, in particular to a method, device and system for dynamically adjusting heartbeat intervals based on node heat. Background technique [0002] The heartbeat mechanism is a key technology to ensure the communication between the NameNode master node and the DataNode slave nodes in the distributed file system HDFS, and to ensure the normal operation of the nodes. The so-called heartbeat mechanism is the mechanism of sending and receiving heartbeat data packets. The slave node periodically sends heartbeat information to the master node actively, and reports the stored data block information of the node and the health status of the node. [0003] The size of the heartbeat interval is configured in the configuration file in advance, and will not be adjusted with changes in node access requirements and data center status. Therefore, for nodes with particularly high access frequency, some access ...

AI Technical Summary

Problems solved by technology

When a DataNode slave node is accessed frequently, the default heartbeat interval will cause some nodes to fail to respond in time and reduce the service performance of the system, resulting in unreal-time update of DataNode slave node information

When a DataNode slave node is accessed very infrequently, the default heartbeat interval causes the NameNode master node to process a large number of high-concurrency useless heartbeat information, which will easily cause a lot of processing pressure on it and increase the number of DataNode The transmission cost sent by the heartbeat packet of the slave node;

[0006] In summary, the traditional periodic heartbeat sending mechanism cannot dynamically adjust the heartbeat sending interval according to the frequency at which each node is visited, that is, the node heat, which is not conducive to improving the performance of the HDFS system.

Images