Client caching method and system based on submodule optimization algorithm

Abstract

The invention discloses a client caching method and system based on a submodule optimization algorithm. The method comprises the following steps: after inputting an access unit by a client, sub-moduleoptimization algorithm processing is carried out to judge whether the access unit needs to be cached or not, an access unit set needing to be cached forms a batch input set in a to-be-cached input set. The caching space is updated according to the data of the batch input set, a cache management mechanism is set on the basis of a three-layer index management unit, a series of operation operators oriented to scenes such as file fragment overlapping, covering and crossing are formulated, and cache units in a complex access mode can be efficiently managed. According to the model, a cache problemis abstracted into a sub-module function optimization problem, the sub-module optimization algorithm is applied to a cache migration strategy, the model provides a synchronous / asynchronous cache replacement / lifting strategy for different application program running modes, and in addition, the model comprises multiple system optimization methods, so that the storage and network communication performance of a client cache is optimized.

Description

technical field [0001] The invention relates to the technical field of distributed file system caching, in particular to a client caching method and system based on a submodel optimization algorithm. Background technique [0002] The upper-layer application program of the distributed file system usually needs to obtain file metadata through RPC communication before using the client of the file system to perform I / O data transmission. However, in scenarios such as frequent random reading of repeated file blocks and repeated reading of small files, this will generate a large amount of RPC communication overhead, which will increase the overall file access time and seriously affect the reading performance. When the file system client corresponding to the application and the process that actually stores data in the file system are located on different nodes, the network overhead caused by remote reading will reduce I / O performance. Even if the data requested by the client proce...

AI Technical Summary

Problems solved by technology

Only relying on server-side caching is difficult to efficiently serve the current emerging application scenarios with frequent access to small-scale data

In addition, existing client caches are organized in units of fixed-size file blocks or objects, and there is currently no related technology and research work to solve the problem of variable-length file fragment caching with complex access patterns including coincidence, coverage, and intersection.

Images