A lightweight approach to autonomous block management for data-intensive file systems

Abstract

The invention discloses an autonomous block management method for a lightweight data-intensive file system. The method is characterized in that the mapping of a data block to a data storage node, quick search of the data block in the data storage node, fast recovery of the data block when the data storage node is in failure, fast redistribution of the data block in the process of newly adding the data storage node and the like are realized by means of ISD (Intersected Shifted Declustering), so that a main node is only in charge of the storage and maintenance of file namespace in the data-intensive file system, but the storage and maintenance of mapping relationship information between the data block and the data storage node, the replacement of the data block when the data storage node is in failure, the redistribution of the data block during the process of adding a new data storage node and the like are all autonomously completed by the data storage node. The method disclosed by the invention has the advantages that memory space of the main node in the data-intensive file system is saved, the processing capability of the main node is improved, and the data block management efficiency of the data-intensive file system under a big-data environment can be dramatically improved.

Description

technical field [0001] The invention relates to computer security technology, in particular to a light-weight autonomous block management method of a data-intensive file system. Background technique [0002] Data-intensive file system DiFS, such as Google File System GFS, Hadoop Distributed File System HDFS, etc., has become the main file system for big data storage management. The current data-intensive file system DiFS adopts a master-slave architecture. The master node (metadata server) manages all metadata, and the slave node (data storage node) is only responsible for data storage. In order to maintain high availability, these storage systems usually divide data files into blocks of fixed size, each data block usually has 3 copies, and distribute them to different data storage nodes of the cluster. The master node must record the addresses of hundreds or thousands of data storage nodes, as well as record the mapping information from the data blocks of all data files to...

AI Technical Summary

Problems solved by technology

[0005] It can be seen from the above analysis that the traditional file system management methods are not suitable for the management of data-intensive file systems. The main reasons are: 1) With the continuous increase of data volume, the storage of file data block address tables will occupy a large amount of storage space ; 2) The master node is responsible for the maintenance of the file data block address table. With the continuous increase of the file data block address table, the processing capacity of the master node is greatly reduced; 3) The continuous increase of the data volume not only takes up a large amount of storage space of the master node , which increases the maintenance cost of metadata such as addresses, and also reduces the scalability of the master node; 4) Each data storage node must first consult the master node when storing and querying, which increases the addressing time

Images