Copy management strategy for data blocks in HDFS

Abstract

A copy management strategy for data blocks in an HDFS comprises a method for selecting the starting time of copies, a method for selecting the number of the copies, a method for selecting the positions of the copies and a method for arranging newly added copies. According to the method for selecting the starting time of the copies, a starting cycle is determined first, one day is regarded as an access cycle to carry out starting of a copy strategy, and then starting moments are determined; according to the method for selecting the number of the copies, the number of the copies are calculated by adopting Poisson distribution, and then calculation is carried out through a copy number calculation method based on a queuing theory; according to the method for selecting the positions of the copies, a copy arranging strategy is adopted in the selection of bays and nodes, and the utilization conditions need to be taken into consideration; according to the method for arranging the newly added copies, the positions of the newly added copies are selected in the bay with the nodes accessed the most by users, and nodes with the lightest load are selected in the bay with the nodes accessed the most by the users to carry out the establishment of the copies. The copy management strategy for the data blocks in the HDFS proposes an HDFS copy-first strategy, can reduce the waiting time when the users access HDFS data and improve the throughput of the system.

Description

technical field [0001] The invention belongs to the technical field of computers, and in particular relates to a copy management strategy of data blocks in HDFS (Hadoop Distributed File System). Background technique [0002] Hadoop is a highly reliable and highly scalable storage and distributed parallel computing platform developed by the Apache open source organization. It was first developed as the basic platform of the open source search engine project Nutch, and then became independent from the Nutch project and became One of the typical open source cloud computing platforms. The Hadoop core implements a block-based distributed file system (Hadoop Distributed File System, HDFS) and a MapReduce computing model for distributed computing. [0003] The HDFS file system uses a block mechanism to store data sets in a distributed manner, and improves system reliability through a data block redundancy strategy. Each data block has multiple copies in the system at the same time...

AI Technical Summary

Problems solved by technology

[0005] However, due to its random nature, this replica placement strategy will lead to uneven placement of data

At the same time, a large amount of data will be generated in the cloud computing environment. These data and their copies are stored in HDFS. Due to the imbalance and uncertainty of users' access to data, some data will be deleted within a certain period of time. Too many visits and become hot data

Since the number of copies of these data is fixed, some access users cannot get a fast response time, causing users to wait indefinitely

Images