Distributed file system and data access method thereof

Abstract

The invention brings forward a distributed file system and a data access method thereof. The system comprises a core control node and multiple sub-clusters. Each sub-cluster comprises a sub-cluster control node and multiple data nodes. The core control node is used for receiving file access requests comprising file names from a client, parsing the file names to determine sub-clusters which files belong to, and transmitting the file access requests to the sub-cluster control nodes in the sub-clusters. The sub-cluster control nodes are used for scheduling the distribution of stored resources of the data nodes in the sub-clusters which the sub-cluster control nodes belong to, establishing indexes to record correlations between the file names of the files stored in the data nodes of the sub-clusters which the sub-cluster control nodes belong to and the data nodes of the files, based on the file access requests transmitted by the core control node and the correlations, determining data nodes related with the file access requests, and informing the client of the identification of the determined data nodes through the forwarding of the core control node.

Description

technical field [0001] The invention belongs to the technical field of distributed storage, in particular to a distributed file system and a data access method thereof. Background technique [0002] Distributed File System (Distributed File System) means that the physical storage resources managed by the file system are not necessarily directly connected to the local node, but are connected to the node through a computer network. The existing distributed file system mainly adopts the client / server mode, and the manager of the file system and the stored data are scattered on different nodes and interconnected through the network. The distributed file system can provide services for multiple clients, allow multiple storage servers to share a global namespace, and realize intelligent distribution of I / O and data to improve system performance. Currently widely used distributed file systems mainly include GFS and Hadoop Distributed File System (HDFS). [0003] GFS is a scalable...

AI Technical Summary

Problems solved by technology

In large-scale applications, there may be thousands of requests at the same time, and it will take a long time to process all these requests, which will inevitably cause a very heavy burden on the NameNode

At the same time, when a single NameNode cluster has a large number of task requests, it cannot respond to load balancing requests in a timely manner. Due to task allocation and other related work, it will also affect the performance of the distributed file system.

[0010] (2) Expansion performance issues

A single NameNode structure will inevitably cause the bottleneck of HDFS system expansion

Images