Distributed multi-level storage system and method based on HDFS

Abstract

The invention provides a distributed multi-level storage system based on HDFS, which uniformly manages and dispatches the data in the multi-level storage device to improve the data storage efficiency.The invention includes realizing policy configuration in the data storage process, wherein the policy configuration comprises data persistence policy configuration, cache data release policy configuration, data locking and unlocking policy configuration, and storage space reservation policy configuration; building and updating data indexes to manage data consistency; data retrieval based on HDFSis realized. By using multi-level storage scheduling technology, data read and write speeds are improved to memory level, it can effectively solve the performance bottleneck of high concurrent data access, improve the system response speed and concurrent processing ability, solve the distributed file system read and write efficiency problems, through thermal data migration, data pre-loading and data persistence functions, effective use of memory and SSD can be achieved, the distributed data read and write speed can be accelerated, the system response speed and concurrent processing ability areimproved.

Description

technical field [0001] The invention relates to the field of distributed storage, in particular to a distributed multi-level storage system and method based on HDFS. Background technique [0002] At present, distributed file systems usually run on general-purpose hardware, and their data is stored in ordinary hard disks. When accessing data with high concurrency, its read and write speed is limited by the hard disk, and there is a performance bottleneck; using solid-state storage disks and memory instead of traditional mechanical hard disks can Improve data read and write efficiency, but in the case of massive data scale, high-performance storage media will bring very huge hardware costs, and system storage has the contradiction between large memory capacity, high speed, and low cost. [0003] Patent document CN104965677A discloses a storage system, which includes computing resource nodes, storage resource nodes, and a PCIe network; wherein, the computing resource nodes and ...

AI Technical Summary

Problems solved by technology

However, the above patent documents do not perform effective resource scheduling on the storage resource nodes of the storage system.

Images