HDFS (Hadoop Distributed File System) intelligent hierarchical storage method and system based on multi-dimensional popularity perception and tape library integration

By building a multi-dimensional heat perception and tape library integration intelligent hierarchical storage method in HDFS, data levels are dynamically divided and cold data is migrated to the tape library. This solves the high cost and resource waste problems of HDFS when storing massive amounts of cold data, and improves system performance and resource utilization.

CN122018813APending Publication Date: 2026-05-12CHANGSHU INSTITUTE OF TECHNOLOGY
0 Cites 1 Cited by

Patent Information

Application Number
CN202610228130.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-02-26
Publication Date
2026-05-12

AI Technical Summary

Technical Problem

Traditional HDFS suffers from high storage costs, low resource utilization, and inability to integrate tape libraries when storing massive amounts of cold data, resulting in wasted storage resources and performance mismatch.

Method used

By constructing a multi-dimensional heat perception and tape library integration HDFS intelligent hierarchical storage method, the heat of data is dynamically perceived, the K-means clustering algorithm is used to divide the data into levels, and the Reed-Solomon RS erasure coding strategy is adopted to migrate cold data to the tape library, and SSD and standard disk are combined for differentiated storage.

Benefits of technology

It significantly reduced storage costs, improved system I/O performance and resource utilization, and achieved efficient data management and storage optimization.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses an HDFS (Hadoop Distributed File System) intelligent hierarchical storage method and system based on multi-dimensional popularity perception and tape library integration, and belongs to the technical field of distributed file systems, and the method comprises the following steps: expanding HDFS metadata to construct a unified namespace; acquiring data block access features and dynamically dividing hot, warm and cold data by using a K-means clustering algorithm; executing a self-adaptive grading strategy: adding a copy to hot data and storing in an SSD (Solid State Disk), maintaining a default copy to warm data and storing in a disk, and migrating cold data to a tape library and storing by adopting an erasure code; scattered storage and fault tolerance of relevance perception are realized through a message queue; metadata persistence and periodical re-evaluation are carried out, and automatic migration and dynamic adjustment of data among the SSD layer, the disk layer and the tape library layer are achieved. By adopting the method and the system, the storage strategy and the data access value are accurately matched, the storage cost is finally reduced, and the system I / O performance and the resource utilization rate are improved.
Need to check novelty before this filing date? Find Prior Art