DFS-oriented client persistent caching method and system in global namespace

Abstract

The invention discloses a DFS-oriented client persistent caching method and system in a global namespace, and belongs to the field of large-scale distributed systems, and the method comprises the steps that a metadata server maintains a view of a global directory tree, all clients share the view, and all the clients can access any file through the view; the client identifies a file needing to be prefetched according to the cache prefetching rule, and migrates the file from the object storage server to the local for persistent caching; when the data volume of the local cache of the client exceeds a cache threshold, the data locally cached by the client is replaced; when the client performs file operation, if the to-be-operated file is cached to the local of the client and the client has thecorresponding operation authority, the locally cached file is directly accessed and the operation is finished; otherwise, the file is remotely accessed in the object storage server through the network and the operation is finished. According to the invention, the data management of the distributed file system can be simplified, and the network resource overhead is reduced.

Description

technical field [0001] The invention belongs to the field of large-scale distributed systems, and more specifically relates to a method and system for persistent caching of a client under a DFS-oriented global namespace. Background technique [0002] In recent years, high-performance computers have widely used large-capacity, high-bandwidth, high-IOPS, mature, stable, and fully functional distributed file systems (Distributed File System, DFS), such as: Lustre, GPFS, etc. In the distributed file system, data and metadata are stored on dedicated server nodes respectively. Specifically, data is stored on the object storage server, and metadata is stored on the metadata server; client nodes (usually computing nodes) pass the network To access the data and metadata shared by the server, this remote access method will cause various problems such as resource contention, I / O congestion, high latency, etc., which will lead to system performance degradation. [0003] The main method...

AI Technical Summary

Problems solved by technology

Under the multi-layer storage of the traditional distributed file system, the data in the cache (such as: Burst Buffer) is temporary and opaque to the application. The prefetch or replacement of the data in the cache is assisted by other tools; the data exists in different storage levels Multiple copies, the consistency of data needs to be coordinated with other tools and the file system to ensure, which will take up a lot of computing and storage resources

In addition, multiple storage layers will use multiple namespaces. The differences in the namespaces of these multiple storage layers make data need to be frequently moved between the buffer layer and the distributed file system, further complicating the workflow

In general, the existing caching methods for distributed file systems require complex data management and consume a large amount of network resources.

Images