HDFS (Hadoop Distributed File System) intelligent hierarchical storage method and system based on multi-dimensional popularity perception and tape library integration

Abstract

The invention discloses an HDFS (Hadoop Distributed File System) intelligent hierarchical storage method and system based on multi-dimensional popularity perception and tape library integration, and belongs to the technical field of distributed file systems, and the method comprises the following steps: expanding HDFS metadata to construct a unified namespace; acquiring data block access features and dynamically dividing hot, warm and cold data by using a K-means clustering algorithm; executing a self-adaptive grading strategy: adding a copy to hot data and storing in an SSD (Solid State Disk), maintaining a default copy to warm data and storing in a disk, and migrating cold data to a tape library and storing by adopting an erasure code; scattered storage and fault tolerance of relevance perception are realized through a message queue; metadata persistence and periodical re-evaluation are carried out, and automatic migration and dynamic adjustment of data among the SSD layer, the disk layer and the tape library layer are achieved. By adopting the method and the system, the storage strategy and the data access value are accurately matched, the storage cost is finally reduced, and the system I / O performance and the resource utilization rate are improved.

Description

Technical Field

[0001] This invention relates to the field of distributed file system technology, and in particular to an HDFS intelligent hierarchical storage method and system based on multi-dimensional heat perception and tape library integration. Background Technology

[0002] With the continuous development of internet technology, we have entered the era of big data, so the application of big data-related technologies should be more in-depth and widespread. The Apache Hadoop framework and its core component HDFS, due to their high throughput, high fault tolerance, and high scalability, have become key infrastructure for processing such big data. However, traditional HDFS faces the following significant drawbacks when dealing with massive amounts of cold data that need to be stored for long periods: First, storage costs are high. HDFS uses a static three-replica strategy by default to ensure data reliability, resulting in a total storage utilization of 300% and additional overhead of up to 200%. For cold data with extremely low access frequency (such as raw experimental data that has already been analyzed), this strategy leads to a huge waste of storage resources. Second, storage resource utilization is low. Although existing HDFS heterogeneous storage research (such as HDFS-2832) supports SSD / HDD tiering, it lacks a fine-grained management mechanism based on data access frequency. This may lead to high-performance storage media (such as SSDs) being occupied by a large amount of cold data, while frequently accessed hot data cannot obtain sufficient performance resources, resulting in resource mismatch. Finally, tape library support is lacking. Tape libraries are an ideal choice for storing massive amounts of cold data due to their advantages such as large storage capacity, low cost, low power consumption, and long data retention period. However, the existing HDFS architecture cannot natively support tape library integration, making it difficult to automatically and transparently archive cold data to tape libraries, and failing to meet the demand for cost-effective archiving of massive amounts of data.

[0003] In summary, there is an urgent need in this field for an HDFS optimization solution that can dynamically sense data popularity, intelligently tier storage, and seamlessly integrate low-cost tape library resources, so as to significantly reduce overall storage costs while ensuring data reliability and access performance. Summary of the Invention

[0004] The purpose of this invention is to provide an intelligent hierarchical storage method and system for HDFS based on multi-dimensional heat perception and tape library integration. This addresses the problems of high storage costs and low resource utilization caused by the lack of tape library support and the single data placement strategy in traditional HDFS heterogeneous storage. By integrating tape libraries and combining multi-dimensional heat perception, intelligent hierarchical storage is achieved, accurately matching storage strategies with data access value, ultimately reducing storage costs and improving system I / O performance and resource utilization.

[0005] To achieve the above objectives, this invention provides an HDFS intelligent hierarchical storage method based on multi-dimensional heat perception and tape library integration, comprising the following steps: S1. Unified Namespace Construction: By extending the metadata management mechanism of the Hadoop Distributed File System (HDFS), the in-memory file metadata structure is redesigned. The storageTierMask attribute is added to the file metadata to record the storage layer where the file is located, and the tapeVolume attribute is added to record the tape volume where the file is located. This allows the tape library to be logically integrated with the standard disk DISK and solid-state drive (SSD) as the ARCHIVE storage layer, and to build a unified in-memory directory tree that provides a transparent file system view. S2. Multi-dimensional heat perception: Under the unified namespace implemented based on the unified memory directory tree, multi-dimensional access characteristics of data blocks in HDFS are collected, including historical access frequency, response time, concurrency, time decay characteristics and data block size. The K-means clustering algorithm is used to dynamically cluster the data blocks and divide the data into three levels: hot data, warm data and cold data. S3. Adaptive hierarchical strategy execution: Based on the three levels of hot data, warm data, and cold data, the corresponding storage strategy is executed. Hot data is retained in 3 copies and stored on SSD media, warm data is retained in 2 copies and stored on standard disk DISK, and cold data is migrated to the tape library ARCHIVE storage layer and stored using the Reed-Solomon RS erasure coding strategy. After the data is successfully written to the tape library, the storageTierMask and tapeVolume attributes of the corresponding file metadata in the unified memory directory tree are updated. S4. Relationship-aware distributed storage: Message queues are used to manage read and write requests to the ARCHIVE storage layer of the tape library, ensuring balanced I / O load during data migration. S5. Metadata Persistence and Lifecycle Management: Persist the unified memory directory tree to the disk image file FsImage and record the operation log EditLog. Ensure the reliability of metadata and rapid recovery after system restart through a periodic checkpoint mechanism. At the same time, set a periodic re-evaluation strategy to realize automatic migration and dynamic adjustment of data between different storage layers.

[0006] Preferably, step S1 specifically includes: S11. By redesigning the in-memory metadata structure of the INodeFile class in HDFS, a unified in-memory directory tree for the hierarchical storage system is constructed, enabling seamless integration of tape layer files in HDFS. S12. Extend the HDFS metadata management mechanism by adding the storageTierMask attribute to the INodeFile class to record storage layer information and the tapeVolume attribute to record tape volume information; the storageTierMask attribute indicates the storage layer where the file copy exists using binary bits. S13. Redesign the file size recording mechanism in the INodeFile class to explicitly record the fileSize attribute for tape layer files; S14. Construct the FSDirectory class to maintain a unified memory directory tree structure, and build the directory hierarchy in memory by maintaining the root directory of type INodeDirectory. S15. Implement a redesigned metadata structure at the HDFS source code level to ensure seamless integration of tape layer file metadata in a unified memory directory tree.

[0007] Preferably, in step S13, the storageTierMask attribute uses a unified binary bit-based notation method, according to...<TIER_SSD_MASK,TIER_DISK_MASK,TIER_TAPE_MASK> Each bit is defined sequentially, with a value of 1 for each bit if a copy exists. The value selection rules are as follows: A value of 1 (binary 001) indicates that the file exists only in the SSD layer (bit 0 is 1, and the rest are 0). A value of 2 (binary 010) indicates that the file exists only in the DISK layer (the first bit is 1, and the rest are 0). A value of 3 (binary 011) indicates that the file has copies in both the SSD and DISK layers (bits 0 and 1 are 1). A value of 4 (binary 100) indicates that the file exists only in the TAPE layer (the second bit is 1, and the rest are 0). A value of 5 (binary 101) indicates that the file has copies in both the SSD and TAPE layers (bits 0 and 2 are 1). A value of 6 (binary 110) indicates that the file has copies in both the DISK and TAPE layers (the first and second bits are 1). A value of 7 (binary 111) indicates that the file has copies on the SSD, DISK, and TAPE layers (bits 0, 1, and 2 are 1). The tapeVolume attribute: When the storageTierMask indicates that the file exists in the TAPE layer (i.e., (storageTierMask&TIER_TAPE_MASK) != 0), the tapeVolume attribute records the unique identifier of the tape volume where the file is located, which is used for tape library positioning.

[0008] Preferably, step S2 specifically includes: S21. Collect multi-dimensional access characteristics of data blocks in HDFS using log collection tools, including: historical access frequency, response time, concurrency, time decay characteristics, and data block size; S22. Analyze multi-dimensional access features using the K-means unsupervised machine learning clustering algorithm, optimize the initial cluster center selection using the K-means++ algorithm, and determine the optimal number of clusters using the elbow rule. S23. Based on the clustering results, the data blocks are divided into three levels: hot data, warm data, and cold data.

[0009] Preferably, step S3 specifically includes: S31. Hot data is stored on SSD media using a 3-replica strategy; S32. Temperature data is stored on a standard disk DISK using a 2-copy strategy. S33. Encode cold data using the Reed-Solomon RS erasure coding algorithm and migrate it to the tape library storage layer.

[0010] Preferably, step S4 includes: S41. Manage request messages to the tape layer through a message server and establish a read / write request queue mechanism; S42. When a tape layer data access failure is detected, automatically switch to the disk layer replica to provide services and trigger the data reconstruction process to establish a tape media health status monitoring and faulty tape isolation mechanism.

[0011] Preferably, step S5 specifically includes: S51. Using the Google Protocol Buffer data serialization protocol, a unified in-memory directory tree containing tape library file metadata information is persisted to an FsImage file; S52. Merge the Editlog and FsImage files using a periodic checkpoint mechanism; S53. Set a periodic re-evaluation strategy and re-execute steps S2 and S3 at preset time intervals.

[0012] This invention also provides an HDFS intelligent tiered storage system based on multi-dimensional heat perception and tape library integration, comprising: The unified namespace building module is used to redesign the in-memory file metadata structure by extending the metadata management mechanism of HDFS, and build a unified in-memory directory tree that provides a transparent file system view. The multi-dimensional heat perception module is used to collect the historical access frequency, response time, concurrency, time decay characteristics and data block size of data blocks in HDFS under a unified namespace based on a unified memory directory tree, and to use the K-means clustering algorithm to dynamically cluster the data blocks, dividing the data into three levels: hot data, warm data and cold data. The adaptive hierarchical strategy execution module is used to execute the corresponding storage strategy according to the three levels. Hot data is retained in 3 copies and stored on SSD media, warm data is retained in 2 copies and stored on standard disk DISK, and cold data is migrated to the tape library ARCHIVE storage layer and stored using the Reed-Solomon RS erasure coding strategy. After the data is successfully written to the tape library, the storageTierMask and tapeVolume attributes of the corresponding file metadata in the unified memory directory tree are updated. The correlation-aware distributed storage module is used to manage read and write requests to the ARCHIVE storage layer of the tape library through message queues, ensuring I / O load balancing during data migration. The metadata persistence and lifecycle management module is used to persist the unified memory directory tree to the disk image file FsImage and record the operation log Editlog. It ensures the reliability of metadata and rapid recovery after system restart through a periodic checkpoint mechanism, and sets a periodic re-evaluation strategy to realize the automatic migration and dynamic adjustment of data between different storage layers.

[0013] Preferably, the system also includes: The exception handling and fault tolerance module is used to automatically switch to the disk layer replica to provide services when tape layer data access fails, and to trigger the data reconstruction process, as well as to establish a tape media health status monitoring and faulty tape isolation mechanism.

[0014] Therefore, this invention employs the aforementioned HDFS intelligent hierarchical storage method and system based on multi-dimensional heat perception and tape library integration. First, it achieves logical integration of the tape library and existing storage media by constructing a unified namespace. Then, it dynamically classifies data heat levels using machine learning clustering algorithms based on multi-dimensional access characteristics. Finally, it implements differentiated storage strategies according to data heat. By migrating cold data to tape libraries employing erasure coding, storage costs are significantly reduced; simultaneously, high-performance storage optimization for hot data improves system I / O performance.

[0015] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0016] Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the construction process of a unified memory directory tree for disk-tape mapping according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating the multi-dimensional heat perception and intelligent hierarchical storage execution process of an embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0018] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0019] Example like Figure 1 As shown, the HDFS intelligent hierarchical storage method based on multi-dimensional heat perception and tape library integration includes the following steps: S1. Unified Namespace Construction: By extending the metadata management mechanism of the Hadoop Distributed File System (HDFS), the in-memory file metadata structure is redesigned. This involves adding the `storageTierMask` attribute to record the storage layer where the file resides and the `tapeVolume` attribute to record the tape volume where the file resides. This logically integrates the tape library as an ARCHIVE storage layer with standard disks and SSDs, constructing a unified in-memory directory tree that provides a transparent file system view. Figure 2 As shown, the specific steps include: S11. By redesigning the in-memory metadata structure of the INodeFile class in HDFS, a unified in-memory directory tree for the hierarchical storage system is constructed, enabling seamless integration of tape layer files in HDFS. S12. Extend the HDFS metadata management mechanism by adding the storageTierMask attribute to the INodeFile class to record storage layer information and the tapeVolume attribute to record tape volume information; the storageTierMask attribute indicates the storage layer where the file copy exists using binary bits. S13. Redesign the file size recording mechanism in the INodeFile class, explicitly recording the fileSize attribute for tape layer files. The storageTierMask attribute uses a unified binary bit-based notation method.<TIER_SSD_MASK,TIER_DISK_MASK,TIER_TAPE_MASK> Each bit is defined sequentially, with a value of 1 for each bit if a copy exists. The value selection rules are as follows: A value of 1 (binary 001) indicates that the file exists only in the SSD layer (bit 0 is 1, and the rest are 0). A value of 2 (binary 010) indicates that the file exists only in the DISK layer (the first bit is 1, and the rest are 0). A value of 3 (binary 011) indicates that the file has copies in both the SSD and DISK layers (bits 0 and 1 are 1). A value of 4 (binary 100) indicates that the file exists only in the TAPE layer (the second bit is 1, and the rest are 0). A value of 5 (binary 101) indicates that the file has copies in both the SSD and TAPE layers (bits 0 and 2 are 1). A value of 6 (binary 110) indicates that the file has copies in both the DISK and TAPE layers (the first and second bits are 1). A value of 7 (binary 111) indicates that the file has copies on the SSD, DISK, and TAPE layers (bits 0, 1, and 2 are 1). The tapeVolume attribute: When the storageTierMask indicates that the file exists in the TAPE layer (i.e., (storageTierMask&TIER_TAPE_MASK) != 0), the tapeVolume attribute records the unique identifier of the tape volume where the file is located, which is used for tape library positioning; S14. Construct the FSDirectory class to maintain a unified memory directory tree structure, and build the directory hierarchy in memory by maintaining the root directory of type INodeDirectory. S15. Implement a redesigned metadata structure at the HDFS source code level to ensure seamless integration of tape layer file metadata in a unified memory directory tree.

[0020] S2. Multi-dimensional Hotness Awareness: Under a unified namespace implemented based on a unified memory directory tree, multi-dimensional access characteristics of data blocks in HDFS are collected, including historical access frequency, response time, concurrency, time decay characteristics, and data block size. The K-means clustering algorithm is used to dynamically cluster the data blocks, dividing the data into three levels: hot data, warm data, and cold data. Figure 3 As shown.

[0021] S21. Collect multi-dimensional access characteristics of data blocks in HDFS using log collection tools, including: historical access frequency, response time, concurrency, time decay characteristics, and data block size; S22. Analyze multi-dimensional access features using the K-means unsupervised machine learning clustering algorithm, optimize the initial cluster center selection using the K-means++ algorithm, and determine the optimal number of clusters using the elbow rule. S23. Based on the clustering results, the data blocks are divided into three levels: hot data, warm data, and cold data.

[0022] S3. Adaptive hierarchical strategy execution: Based on the three levels of hot data, warm data, and cold data, the corresponding storage strategy is executed. Hot data is retained in 3 copies and stored on SSD media, warm data is retained in 2 copies and stored on standard disk DISK, and cold data is migrated to the tape library ARCHIVE storage layer and stored using the Reed-Solomon RS erasure coding strategy. After the data is successfully written to the tape library, the storageTierMask and tapeVolume attributes of the corresponding file metadata in the unified memory directory tree are updated. S31. Hot data is stored on SSD media using a 3-replica strategy; S32. Temperature data is stored on a standard disk DISK using a 2-copy strategy. S33. Encode the cold data using the Reed-Solomon RS erasure coding algorithm (RS-6-3 encoding strategy) and migrate it to the tape library storage layer.

[0023] S4. Relationship-aware distributed storage: Message queues are used to manage read and write requests to the ARCHIVE storage layer of the tape library, ensuring balanced I / O load during data migration. S41. Manage request messages to the tape layer through a message server and establish a read / write request queue mechanism; S42. When a tape layer data access failure is detected, automatically switch to the disk layer replica to provide services and trigger the data reconstruction process to establish a tape media health status monitoring and faulty tape isolation mechanism.

[0024] S5. Metadata Persistence and Lifecycle Management: Persist the unified memory directory tree to the disk image file FsImage and record the operation log Editlog. Ensure the reliability of metadata and rapid recovery after system restart through a periodic checkpoint mechanism. At the same time, set a periodic re-evaluation strategy to realize automatic migration and dynamic adjustment of data between different storage layers.

[0025] S51. Using the Google Protocol Buffer data serialization protocol, a unified in-memory directory tree containing tape library file metadata information is persisted to an FsImage file; S52. Merge the EditLog and FsImage files using a periodic checkpoint mechanism; S53. Set a periodic re-evaluation strategy and re-execute steps S2 and S3 at preset time intervals.

[0026] This embodiment also provides an HDFS intelligent tiered storage system based on multi-dimensional heat perception and tape library integration, including: The unified namespace building module is used to redesign the in-memory file metadata structure by extending the metadata management mechanism of HDFS, and build a unified in-memory directory tree that provides a transparent file system view. The multi-dimensional heat perception module is used to collect the historical access frequency, response time, concurrency, time decay characteristics and data block size of data blocks in HDFS under a unified namespace based on a unified memory directory tree, and to use the K-means clustering algorithm to dynamically cluster the data blocks, dividing the data into three levels: hot data, warm data and cold data. The adaptive hierarchical strategy execution module is used to execute the corresponding storage strategy according to the three levels. Hot data is retained in 3 copies and stored on SSD media, warm data is retained in 2 copies and stored on standard disk DISK, and cold data is migrated to the tape library ARCHIVE storage layer and stored using the Reed-Solomon RS erasure coding strategy. After the data is successfully written to the tape library, the storageTierMask and tapeVolume attributes of the corresponding file metadata in the unified memory directory tree are updated. The correlation-aware distributed storage module is used to manage read and write requests to the ARCHIVE storage layer of the tape library through message queues, ensuring I / O load balancing during data migration. The metadata persistence and lifecycle management module is used to persist the unified memory directory tree to the disk image file FsImage and record the operation log EditLog. It ensures the reliability of metadata and rapid recovery after system restart through a periodic checkpoint mechanism, and sets a periodic re-evaluation strategy to realize the automatic migration and dynamic adjustment of data between different storage layers.

[0027] The system also includes: The exception handling and fault tolerance module is used to automatically switch to the disk layer replica to provide services when tape layer data access fails, and to trigger the data reconstruction process, as well as to establish a tape media health status monitoring and faulty tape isolation mechanism.

[0028] Example 1 This embodiment details the specific implementation of the HDFS intelligent hierarchical storage method based on multi-dimensional heat perception and tape library integration.

[0029] Step A: Building a unified namespace.

[0030] This step forms the foundation for implementing intelligent tiered storage in HDFS. Its core lies in extending the HDFS NameNode's in-memory metadata structure to build a logically seamless unified storage namespace. The specific implementation includes the following sub-steps: A1: Defines the storage layer bitmask.

[0031] Define and implement a storage tier mask (storageTierMask) for encoding the physical distribution status of files. This mask is an integer field, where each bit represents a specific storage tier. In this embodiment, it is defined as follows: The 0th bit (TIER_SSD_MASK, value 1) represents the SSD storage layer.

[0032] The first position (TIER_DISK_MASK, value 2) represents the standard disk (DISK) storage layer.

[0033] The second position (TIER_TAPE_MASK, value 4) represents the archive / tape storage layer.

[0034] A2: Extended file metadata attributes.

[0035] The HDFS INodeFile class has been redesigned, adding the following key properties: storageTierMask: Records the storage layer where file copies exist. When a valid copy of a file exists in a certain storage layer, the corresponding bit is set to 1. For example, if a file exists in both disk and tape layers, then its storageTierMask = TIER_DISK_MASK|TIER_TAPE_MASK = 2|4 = 6.

[0036] tapeVolume: When the storageTierMask indicates that the file exists on a tape layer, this attribute records the unique identifier of the tape volume where the file resides.

[0037] tapeFileOffset: When a file exists on a tape layer, this attribute records the logical starting position of the file within the tape volume.

[0038] `fileSize`: Explicitly records the logical size of the file. This attribute is crucial for tape layer files because tape files do not follow the standard HDFS block model and their size cannot be dynamically calculated from data block information.

[0039] A3: Construct a unified memory directory tree.

[0040] Based on the extended INodeFile class, a unified in-memory directory tree is constructed for the hierarchical storage system. The root directory rootDir (of type INodeDirectory) is maintained by the FSDirectory class, thus building the directory hierarchy in memory. In this structure, a hot data file stored on the SSD and a cold data file archived to the tape library are represented as equal INodeFile objects in the unified in-memory directory tree, sharing the same namespace path, permission attributes, and parent-child relationships.

[0041] A4: Implement storage state updates.

[0042] Based on the unified metadata architecture described above, atomic operations for data migration are achieved. When the system decides to migrate a file from disk to tape, the following atomic operations are performed: Write the file data to the specified tape volume and obtain tapeVolume and tapeFileOffset.

[0043] The metadata of the file INodeFile is updated atomically: the TIER_TAPE_MASK bit in storageTierMask is set, and tapeVolume and tapeFileOffset are written. Depending on the policy, disk-level copies may be selectively wiped (i.e., the TIER_DISK_MASK bit in storageTierMask is cleared).

[0044] Conversely, when it is necessary to retrieve (restore) a tape file, the decision-making process queries its physical coordinates based on the same set of metadata, thereby triggering an asynchronous retrieval task.

[0045] Through the above steps, a true logical unification from the user's perspective to the system management perspective is achieved, providing precise metadata support and a control foundation for subsequent multi-dimensional heat perception and intelligent grading strategies. The core process of this method can be found in [reference needed]. Figure 3 The data archiving and retrieval decisions both rely on the unified namespace built in this step.

[0046] Step B: Multi-dimensional heat perception and intelligent classification.

[0047] Based on a unified memory directory tree, data popularity is accurately and dynamically perceived and a hierarchical strategy is implemented.

[0048] B1: Multi-dimensional access feature collection.

[0049] The system continuously collects fine-grained access characteristics for each data block, constructing a feature vector describing the data access characteristics, including: Access frequency: The number of read operations within the statistical time window.

[0050] Response time: The average response time for accessing recorded data blocks.

[0051] Concurrency: Monitors the number of clients accessing this data block simultaneously.

[0052] Time decay feature: This is an indicator used to measure the access frequency or timeliness of data blocks. Its core idea is to reflect the "freshness" or "importance decaying over time" of data by calculating the impact of the time elapsed since the last access to the data block on its current value or priority.

[0053] Data block size: Records the physical size information of the data block.

[0054] B2: K-Means clustering algorithm execution.

[0055] The K-Means++ algorithm is used to optimize the initial cluster center selection, and the optimal number of clusters K=3 (corresponding to the three levels of "hot", "warm", and "cold") is determined by the elbow rule. The clustering process minimizes the following loss function: ; in, It is the first A cluster; It is the feature vector of the data block; It is the first Cluster centers of each cluster This indicates the number of clusters, which is the number of different groups (clusters) into which the data is divided. The loss function value, also known as the cost function value, represents the value used in the clustering process. It measures the quality of the clustering results; a smaller loss function value indicates a better clustering result, meaning the data blocks are more compactly distributed within the clusters. The norm of a vector, specifically the Euclidean norm (also known as the L2 norm) in this paper, is calculated by summing the sums of the squared distances across all data blocks within each cluster, and then summing the sums across all clusters. This yields the clustering loss for the entire dataset. The smaller this sum of squared distances, the more concentrated the distribution of data blocks within the clusters, and the better the clustering effect.

[0056] B3: Dynamic popularity level classification.

[0057] Based on the clustering results, the data blocks are divided into three levels: Hot data: Feature vectors stand out in the "high frequency, short interval" dimension, clustering into highly active clusters.

[0058] Warm data: Feature vectors exhibit moderate performance in accessing features, forming clusters with moderate activity.

[0059] Cold data: Feature vectors are prominent in the "low frequency, long interval" dimension, forming low-activity clusters.

[0060] B4: Adaptive storage strategy execution.

[0061] Implement differentiated storage strategies based on popularity levels: Hot data processing strategy: For data identified as "hot", the system maintains its 3-replica storage strategy and prioritizes placing these replicas on SSD media.

[0062] Warm data processing strategy: For data identified as "warm", the system uses a 2-copy storage strategy to store it on a standard disk (DISK).

[0063] Cold data processing strategy: For data identified as "cold," the system automatically migrates it to the tape library archiving layer, where it is protected using Reed-Solomon erasure coding (this embodiment uses RS(6,3) encoding; Reed-Solomon (RS) erasure coding is a widely used error correction coding technique, but it is not the only option. In practical applications, the choice of encoding method depends on specific needs, such as storage efficiency, computational complexity, and error recovery capabilities). This significantly reduces storage overhead from 300% under the three-copy strategy to approximately 150%, as shown in Table 1.

[0064] Table 1. Correlation Table of HDFS Tiered Storage Benefits

[0065] After the migration is complete, the storageTierMask, tapeVolume and other attributes in the file metadata are updated, and the system can selectively delete the original copies of the online layer according to the policy, further freeing up expensive online storage resources.

[0066] B5: Periodic reassessment.

[0067] Cluster analysis is re-executed at fixed time intervals (e.g., weekly) to dynamically update cluster centers. This ensures that the popularity ranking matches the current actual access patterns, enabling continuous management of the data lifecycle.

[0068] Step C: Relevance-aware distributed storage.

[0069] To address the high latency of sequential access to magnetic tape media and ensure I / O load balancing during data migration, the system implements an asynchronous task management mechanism.

[0070] C1: Request analysis and message encapsulation.

[0071] Monitor all access intents to the ARCHIVE storage layer of the tape library. When such a request is detected, encapsulate it into a standardized task message containing information such as task ID, operation type, target file identifier, target tape volume identifier, and task priority.

[0072] C2: Message submission and queue buffering.

[0073] Submit the task message to a separate, highly available message queue service (e.g., built on RabbitMQ or Apache Kafka). This step transforms synchronous, long-latent I / O requests into asynchronous tasks buffered in the message queue, smoothing out peak traffic and supporting priority handling.

[0074] C3: Backend service consumption and task scheduling.

[0075] The backend tape drive service (Tape Server) acts as a consumer, pulling tasks from the message queue according to a strategy (such as FIFO, or a scheduling algorithm combined with priority). Internally, the Tape Server maintains a task executor, which parses task messages and calls the corresponding tape library operation instructions according to the operation type, optimizing the execution order to reduce mechanical operation time.

[0076] C4: Data prefetching / writing and cache management (core I / O operations).

[0077] For data read (prefetch) requests: The Tape Server controls the tape library robot to load the specified tape volume according to the task instructions, locates the file logical offset (tapeFileOffset), and reads the data sequentially from the tape into the pre-allocated, high-speed disk cache (Staging Area).

[0078] For data write (archiving / migration) requests: Tape Server writes data from the source location (such as disk data blocks to be migrated) to the tape library and updates the file's metadata (such as setting the storageTierMask property, recording tapeVolume and tapeFileOffset) upon successful completion.

[0079] Cache Management: The system needs to implement a disk cache space management strategy, such as the LRU (Least Recently Used) eviction algorithm, to ensure efficient utilization of cache space. It should also record the mapping relationship between cached data and tape files.

[0080] C5: Status updates and client notifications.

[0081] After the task is executed successfully, the Tape Server updates the task status to "Completed"; For read requests, notify the client that the data is ready in the disk cache path; For write requests, notify the relevant parties that the task is complete.

[0082] C6: Exception handling and retry mechanism.

[0083] Establish a fault tolerance mechanism. When a task fails (e.g., tape read error, drive failure), the Tape Server marks the task status as "failed" and can retry a limited number of times according to a preset strategy (e.g., exponential backoff). If the task still fails after retrying, it is transferred to a dead-letter queue for further processing by the administrator.

[0084] Step D: Metadata persistence and dynamic re-evaluation.

[0085] To ensure the recoverability of system status and the adaptability of hierarchical strategies, a full lifecycle metadata management and dynamic adjustment mechanism should be established.

[0086] D1: Persistence of metadata changes.

[0087] Any operation that causes a change in the state of the data block storage layer is written as a transaction record in the EditLog operation log.

[0088] D2: Create checkpoints periodically.

[0089] The system periodically serializes the complete unified memory directory tree in memory using the NameNode's checkpoint process and persists it to a new FsImage image file using the Google Protocol Buffer data serialization protocol. Subsequently, the current EditLog is merged with the new FsImage.

[0090] D3: The system status is recoverable.

[0091] When the system restarts, it first loads the latest FsImage file to rebuild the metadata directory tree, and then replays the subsequent EditLog files to accurately restore the metadata state to the last consistent state.

[0092] D4: Set a fixed re-evaluation cycle and trigger the perception process.

[0093] The system establishes a fixed reassessment cycle (e.g., every 7 days). When the cycle is reached, a new round of the complete data heat perception and classification strategy execution process is automatically triggered, that is, step B is re-executed.

[0094] D5: Dynamic data migration based on new perception results.

[0095] Based on the newly generated heat clustering results, the system automatically makes decisions and executes the migration of data between different storage layers, forming a closed, adaptive data lifecycle management loop.

[0096] Therefore, this invention employs the aforementioned HDFS intelligent hierarchical storage method and system based on multi-dimensional heat perception and tape library integration. First, it achieves logical integration of the tape library and existing storage media by constructing a unified namespace. Then, it dynamically classifies data heat levels using machine learning clustering algorithms based on multi-dimensional access characteristics. Finally, it implements differentiated storage strategies according to data heat. By migrating cold data to tape libraries employing erasure coding, storage costs are significantly reduced; simultaneously, high-performance storage optimization for hot data improves system I / O performance. Experimental results show that this method achieves the optimal balance between storage cost and access performance while ensuring data reliability.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An HDFS intelligent hierarchical storage method based on multi-dimensional heat perception and tape library integration, characterized in that, Includes the following steps: S1. Unified Namespace Construction: By extending the metadata management mechanism of the Hadoop Distributed File System (HDFS), the in-memory file metadata structure is redesigned. The storageTierMask attribute is added to the file metadata to record the storage layer where the file is located, and the tapeVolume attribute is added to record the tape volume where the file is located. This allows the tape library to be logically integrated with the standard disk DISK and solid-state drive (SSD) as the ARCHIVE storage layer, and to build a unified in-memory directory tree that provides a transparent file system view. S2. Multi-dimensional heat perception: Under the unified namespace implemented based on the unified memory directory tree, multi-dimensional access characteristics of data blocks in HDFS are collected, including historical access frequency, response time, concurrency, time decay characteristics and data block size. The K-means clustering algorithm is used to dynamically cluster the data blocks and divide the data into three levels: hot data, warm data and cold data. S3. Adaptive hierarchical strategy execution: Based on the three levels of hot data, warm data, and cold data, the corresponding storage strategy is executed. Hot data is retained in 3 copies and stored on SSD media, warm data is retained in 2 copies and stored on standard disk DISK, and cold data is migrated to the tape library ARCHIVE storage layer and stored using the Reed-Solomon RS erasure coding strategy. After the data is successfully written to the tape library, the storageTierMask and tapeVolume attributes of the corresponding file metadata in the unified memory directory tree are updated. S4. Relationship-aware distributed storage: Message queues are used to manage read and write requests to the ARCHIVE storage layer of the tape library, ensuring balanced I / O load during data migration. S5. Metadata Persistence and Lifecycle Management: Persist the unified memory directory tree to the disk image file FsImage and record the operation log EditLog. Ensure the reliability of metadata and rapid recovery after system restart through a periodic checkpoint mechanism. At the same time, set a periodic re-evaluation strategy to realize automatic migration and dynamic adjustment of data between different storage layers.

2. The HDFS intelligent hierarchical storage method based on multi-dimensional heat perception and tape library integration as described in claim 1, characterized in that, Step S1 specifically includes: S11. By redesigning the in-memory metadata structure of the INodeFile class in HDFS, a unified in-memory directory tree for the hierarchical storage system is constructed, enabling seamless integration of tape layer files in HDFS. S12. Extend the HDFS metadata management mechanism by adding the storageTierMask attribute to the INodeFile class to record storage layer information and the tapeVolume attribute to record tape volume information; the storageTierMask attribute indicates the storage layer where the file copy exists using binary bits. S13. Redesign the file size recording mechanism in the INodeFile class to explicitly record the fileSize attribute for tape layer files; S14. Construct the FSDirectory class to maintain a unified memory directory tree structure, and build the directory hierarchy in memory by maintaining the root directory of type INodeDirectory. S15. Implement a redesigned metadata structure at the HDFS source code level to ensure seamless integration of tape layer file metadata in a unified memory directory tree.

3. The HDFS intelligent hierarchical storage method based on multi-dimensional heat perception and tape library integration as described in claim 2, characterized in that, In step S13, the storageTierMask attribute uses a unified binary bit-based notation method, according to...<TIER_SSD_MASK,TIER_DISK_MASK,TIER_TAPE_MASK> Each bit is defined sequentially, with a value of 1 for each bit if a copy exists. The value selection rules are as follows: A value of 1 indicates that the file exists only on the SSD layer. A value of 2 indicates that the file exists only in the DISK layer. A value of 3 indicates that the file has copies in both the SSD and DISK layers. A value of 4 indicates that the file only exists in the TAPE layer. A value of 5 indicates that the file has copies in both the SSD and TAPE layers. A value of 6 indicates that the file has copies in both the DISK and TAPE layers; A value of 7 indicates that the file has copies on three levels: SSD, DISK, and TAPE. The tapeVolume attribute: When the storageTierMask file exists in the TAPE layer, the tapeVolume attribute records the unique identifier of the tape volume where the file is located, which is used for tape library positioning.

4. The HDFS intelligent hierarchical storage method based on multi-dimensional heat perception and tape library integration according to claim 1, characterized in that, Step S2 specifically includes: S21. Collect multi-dimensional access characteristics of data blocks in HDFS using log collection tools, including: historical access frequency, response time, concurrency, time decay characteristics, and data block size; S22. Analyze multi-dimensional access features using the K-means unsupervised machine learning clustering algorithm, optimize the initial cluster center selection using the K-means++ algorithm, and determine the optimal number of clusters using the elbow rule. S23. Based on the clustering results, the data blocks are divided into three levels: hot data, warm data, and cold data.

5. The HDFS intelligent hierarchical storage method based on multi-dimensional heat perception and tape library integration according to claim 1, characterized in that, Step S3 specifically includes: S31. Hot data is stored on SSD media using a 3-replica strategy; S32. Temperature data is stored on a standard disk DISK using a 2-copy strategy. S33. Encode cold data using the Reed-Solomon RS erasure coding algorithm and migrate it to the tape library storage layer.

6. The HDFS intelligent hierarchical storage method based on multi-dimensional heat perception and tape library integration according to claim 1, characterized in that, Step S4 includes: S41. Manage request messages to the tape layer through a message server and establish a read / write request queue mechanism; S42. When a tape layer data access failure is detected, automatically switch to the disk layer replica to provide services and trigger the data reconstruction process to establish a tape media health status monitoring and faulty tape isolation mechanism.

7. The HDFS intelligent hierarchical storage method based on multi-dimensional heat perception and tape library integration according to claim 1, characterized in that, Step S5 specifically includes: S51. Using the Google Protocol Buffer data serialization protocol, a unified in-memory directory tree containing tape library file metadata information is persisted to an FsImage file; S52. Merge the EditLog and FsImage files using a periodic checkpoint mechanism; S53. Set a periodic re-evaluation strategy and re-execute steps S2 and S3 at preset time intervals.

8. An HDFS intelligent hierarchical storage system based on multi-dimensional heat perception and tape library integration, used to execute the HDFS intelligent hierarchical storage method based on multi-dimensional heat perception and tape library integration as described in any one of claims 1-7, characterized in that, include: The unified namespace building module is used to redesign the in-memory file metadata structure by extending the metadata management mechanism of HDFS, and build a unified in-memory directory tree that provides a transparent file system view. The multi-dimensional heat perception module is used to collect the historical access frequency, response time, concurrency, time decay characteristics and data block size of data blocks in HDFS under a unified namespace based on a unified memory directory tree, and to use the K-means clustering algorithm to dynamically cluster the data blocks, dividing the data into three levels: hot data, warm data and cold data. The adaptive hierarchical strategy execution module is used to execute the corresponding storage strategy according to the three levels. Hot data is retained in 3 copies and stored on SSD media, warm data is retained in 2 copies and stored on standard disk DISK, and cold data is migrated to the tape library ARCHIVE storage layer and stored using the Reed-Solomon RS erasure coding strategy. After the data is successfully written to the tape library, the storageTierMask and tapeVolume attributes of the corresponding file metadata in the unified memory directory tree are updated. The correlation-aware distributed storage module is used to manage read and write requests to the ARCHIVE storage layer of the tape library through message queues, ensuring I / O load balancing during data migration. The metadata persistence and lifecycle management module is used to persist the unified memory directory tree to the disk image file FsImage and record the operation log EditLog. It ensures the reliability of metadata and rapid recovery after system restart through a periodic checkpoint mechanism, and sets a periodic re-evaluation strategy to realize the automatic migration and dynamic adjustment of data between different storage layers.

9. The HDFS intelligent hierarchical storage system based on multi-dimensional heat perception and tape library integration according to claim 8, characterized in that, The system also includes: The exception handling and fault tolerance module is used to automatically switch to the disk layer replica to provide services when tape layer data access fails, and to trigger the data reconstruction process, as well as to establish a tape media health status monitoring and faulty tape isolation mechanism.

Images