Metadata query efficiency improving method and system based on BeeGFS

Abstract

The invention discloses a BeeGFS-based metadata query efficiency improvement method and system. The BeeGFS-based metadata query efficiency improvement method comprises the following steps: constructing a three-dimensional index network consisting of a PMEMKV hot data layer, a RocksDB common data layer and a ZFS cold data layer; dividing all metadata into three independent sets of hot data, common data and cold data according to the heat values, and respectively storing the three independent sets to corresponding layers; after a query request is received, a top layer, a middle layer and a bottom layer are queried in sequence, and corresponding metadata is returned if the query request is hit; monitoring query indexes in real time, wherein the query indexes are used for updating the heat value of the metadata and adjusting the storage configuration of each level; performing dynamic promotion or degradation on the metadata between the three-dimensional metadata index networks according to the latest popularity value of the metadata and a preset popularity threshold value, and performing data synchronization and query path switching in the promotion or degradation process; the method can reduce query delay, improve throughput and balance storage resources.

Description

Technical Field

[0001] This invention relates to computer storage, and in particular to a method and system for improving metadata query efficiency based on BeeGFS. Background Technology

[0002] In the era of integrated high-performance computing and big data applications, the data storage scale of supercomputing centers has exceeded the EB level, exhibiting characteristics of "massive small file volume, complex access patterns, and real-time query requirements." Metadata, as data describing key information such as data attributes, location, and access permissions, directly determines the response speed of the entire storage system through its query efficiency, making it a core bottleneck affecting business processing efficiency.

[0003] BeeGFS, as a typical distributed parallel file system, is widely used in the HPC (High-Performance Computing) field due to its scalability; however, its native metadata management architecture has significant shortcomings in query efficiency.

[0004] High query latency: Using a local file system (such as XFS) to store metadata, each query requires a complete call chain from the VFS layer to the block device driver. In the read alignment stage of gene sequencing (which requires high-frequency metadata queries), the latency of a single query can reach hundreds of microseconds, resulting in significant performance loss.

[0005] Directory traversal is inefficient: It adopts a flat storage structure and stores directory items in the form of linked lists. In scenarios with multi-level nested directories and a large number of file sets, directory traversal requires traversing the linked list nodes one by one. For example, in the initialization stage of weather simulation, tens of thousands of files need to be traversed, which takes up to seconds.

[0006] Storage resources are mismatched with query efficiency: There is a lack of dynamic hierarchical storage mechanism. Hot data (high-frequency queries) and cold data (low-frequency queries) share storage resources. Hot data cannot obtain dedicated high-performance storage support, while cold data occupies high-performance storage resources, which dilutes query resources. During the dataset loading stage of AI training (where hot data accounts for only 10%-20%), the problems of low query hit rate and slow response are particularly prominent.

[0007] Existing optimization solutions have significant limitations: Lustre file system's MDS cluster mode is constrained by static mapping of metadata partitions, requiring partition reallocation after adding nodes, resulting in poor query load balancing and large fluctuations in query latency after expansion; the RADOS Gateway solution for distributed storage systems introduces additional network transmission overhead, increasing metadata query latency by more than 30% compared to the native solution; BeeGFS community edition's metadata cache only supports a single-node LRU strategy, lacking cluster-level collaboration, preventing efficient sharing of hot data across the cluster, and leading to low cross-node query efficiency. Therefore, there is an urgent need to build a new metadata indexing architecture to comprehensively optimize query efficiency from dimensions such as storage hierarchy, index design, and data flow. Summary of the Invention

[0008] Purpose of the invention: The purpose of this invention is to provide a method and system for improving metadata query efficiency based on BeeGFS, which can reduce query latency, increase throughput, and balance storage resources.

[0009] Technical solution: The method for improving metadata query efficiency based on BeeGFS described in this invention includes the following processes:

[0010] A three-dimensional metadata index network is constructed, which specifically includes a top-level index using the PMEMKV hot data index cluster, an intermediate-level index using the RocksDB distributed metadata storage, and a bottom-level index using the ZFS cold data archiving system.

[0011] All metadata is divided into three independent sets according to popularity value: hot data, frequently used data, and cold data, and stored in the top-level index, intermediate-level index, and bottom-level index respectively.

[0012] Upon receiving a query request, the system queries the top-level index, intermediate-level index, and bottom-level index in sequence. If a match is found, the corresponding metadata is returned; otherwise, a null value is returned directly.

[0013] Real-time monitoring of query metrics at each level is used to update metadata popularity values ​​and load status at each level in real time; storage configuration at each level is adjusted based on the latest popularity values ​​and load status.

[0014] Based on the latest popularity value and preset popularity threshold, metadata is dynamically promoted or downgraded across the three-dimensional metadata index network. The promotion or downgrade process involves synchronizing data first and then switching query paths.

[0015] By classifying metadata into three levels—"hot," "frequently used," and "cold"—and storing them in index layers with decreasing performance (PMEMKV, RocksDB, and ZFS), compared to the flat architecture of native BeeGFS which stores all metadata indiscriminately in a single local file system (such as XFS), this approach achieves a precise match between storage resources and access demands. It balances storage resources, ensuring that the pressure of high-concurrency queries is mainly borne by the high-performance top and middle layers, avoiding slow disks becoming a system bottleneck, and ensuring query efficiency.

[0016] The order of querying the top, middle, and bottom layers sequentially creates a path where performance decreases progressively, but so does the query cost. The vast majority of online business requests (for frequently accessed and frequently used data) can be responded to quickly in the first two layers, with only a very small number of low-frequency queries requiring a penetration to the bottom layer. This systematically reduces overall query latency and improves the user experience.

[0017] By monitoring query metrics at each level and updating and adjusting relevant data, a closed-loop feedback control system is formed, enabling the system to adapt to changes in business load. Specifically, through "dynamic promotion or demotion," the data distribution can be automatically adjusted according to changes in query status, ensuring that high-performance resources always serve the most needed data. Furthermore, the "synchronize first, then switch" approach during data migration avoids reducing query speed.

[0018] Preferably, the specific architecture of each layer of the three-dimensional metadata indexing network includes:

[0019] Top-level index: A distributed index cluster is built using persistent memory as the storage medium and PMEMKV as the storage engine;

[0020] Intermediate tier index: NVMe-SSDs are used as the storage medium and RocksDB as the storage engine to build a sharded distributed storage cluster;

[0021] Underlying index: An archive storage system is built using SATA-SSD and mechanical hard drive as storage media and ZFS as the file system.

[0022] Top-level hot data utilizes Persistent Memory (PMEM) and the PMEMKV engine, fully leveraging PMEM's byte-addressing, low latency, and high bandwidth characteristics, combined with PMEMKV's key-value storage optimization, laying the physical foundation for achieving microsecond-level query response times. Middle-level frequently used data uses NVMe SSDs and the RocksDB engine, utilizing NVMe SSDs' high IOPS and RocksDB's high write throughput and good read performance to ensure high throughput and millisecond-level response times for complex queries. Bottom-level cold data uses SATA SSDs + HDDs and the ZFS file system, employing ZFS's advanced features such as data integrity, snapshots, and compression to achieve highly reliable and queryable archive storage at a lower cost. Each layer employs a different architecture, allowing each layer to be independently optimized, expanded, and maintained.

[0023] Preferably, the formula for calculating the heat value is as follows:

[0024]

[0025] Where H is the heat value, , and These are the weighting coefficients. Here, T represents the metadata access frequency, T represents the interval between the most recent access time of the metadata and the current time, and B represents the average bandwidth of a single metadata access.

[0026] Compared to the traditional LRU strategy based solely on access frequency (F), the heat value calculation formula provided by this invention introduces a time decay factor (F). ) and bandwidth consumption factor (B), By capturing the temporal locality of data that is "more likely to be accessed again after recent access", the system can respond quickly to sudden access. The bandwidth consumption factor B can identify which data may trigger a large number of I / O operations (such as large file metadata) in a single access. Even if the access frequency is not high, it may become a system bottleneck. Therefore, it needs to be given a higher heat weight. Thus, the heat value calculated by the three-dimensional model provided by this invention is more comprehensive and reliable.

[0027] Preferably, the promotion process is triggered when H ≥ 2.0, and the demotion process is triggered when H ≤ 0.1.

[0028] After the promotion process is triggered, further evaluation is required. Promotion can only proceed if the following conditions are met:

[0029] When metadata in the underlying index is promoted to the intermediate index, H must be ≥ 1.5 for 5 consecutive minutes.

[0030] When metadata from an intermediate-level index is promoted to the top-level index, it must meet the condition that H ≥ 2.0 for 3 consecutive minutes.

[0031] After the downgrade process is triggered, further judgment is required. The downgrade operation can only proceed if the following conditions are met:

[0032] When metadata in a top-level index is downgraded to an intermediate-level index, H must be less than or equal to 0.3 for one consecutive hour.

[0033] When intermediate-level index metadata is downgraded to the underlying index, H must be less than or equal to 0.1 for 24 consecutive hours.

[0034] By requiring the popularity value to remain consistently above or below a threshold over a continuous period (5 minutes, 3 minutes, 1 hour, or 24 hours), rather than triggering migration upon instantaneous reaching of the threshold, this design effectively avoids frequent and unnecessary back-and-forth migration of metadata between storage levels due to short-term fluctuations or instantaneous spikes in query traffic. This "steady-state migration" strategy significantly reduces network bandwidth, CPU, and I / O overhead caused by data migration itself, ensuring the performance stability of the core query service.

[0035] The BeeGFS-based metadata query efficiency improvement system of the present invention includes:

[0036] Network building module: used to build a three-dimensional metadata index network, specifically including a top-level index using the PMEMKV hot data index cluster, an intermediate-level index using the RocksDB distributed metadata storage, and a bottom-level index using the ZFS cold data archiving system;

[0037] Metadata classification module: This module is used to divide all metadata into three independent sets based on popularity: hot data, frequently used data, and cold data, and store them in the top-level index, intermediate-level index, and bottom-level index respectively.

[0038] Monitoring module: Used to monitor query metrics at each level in real time, update metadata popularity values ​​and load status at each level in real time; adjust storage configuration at each level based on the latest popularity values ​​and load status at each level.

[0039] Data migration module: used to dynamically promote or downgrade metadata across the 3D metadata index network based on the latest popularity value and preset popularity threshold. The promotion or downgrade process first synchronizes the data and then switches the query path.

[0040] Data migration module: used to dynamically promote or downgrade metadata across the 3D metadata index network based on the latest popularity value and preset popularity threshold. The promotion or downgrade process first synchronizes the data and then switches the query path.

[0041] Beneficial Effects: By constructing a three-dimensional heterogeneous index network, a physical foundation for resource tiered optimization was laid; through multi-dimensional hot data quantitative calculation and steady-state migration rules, precise and smooth data scheduling was achieved; through optimization of the PMEMKV top layer in terms of data structure, concurrency control, cluster management, and persistence, stable support for microsecond-level high-concurrency queries of hot data was achieved; through in-depth tuning of the RocksDB middleware layer in terms of distributed architecture, multi-dimensional indexes, cache prefetching, and storage engine parameters, high throughput and complex query capabilities were achieved; through the design of intelligent compression, secondary indexes, and recall mechanisms at the ZFS underlying layer, low-cost, reliable, and efficient archiving management of cold data was achieved. This series of interconnected technical measures worked synergistically to achieve significant comprehensive benefits, including reduced query latency, increased concurrent throughput, improved efficiency in directory traversal and complex queries, increased storage resource utilization, and reduced overall costs. It systematically solved the query performance bottleneck problem of EB-level massive metadata in demanding scenarios such as high-performance computing and artificial intelligence. Attached Figure Description

[0042] Figure 1 This is a diagram illustrating the overall architecture of the 3D metadata index.

[0043] Figure 2 Top-level index - PMEMKV hot data index architecture diagram;

[0044] Figure 3 This is a diagram of the RocksDB distributed metadata storage architecture in the middle layer.

[0045] Figure 4 This is a diagram of the ZFS cold data archiving system architecture. Detailed Implementation

[0046] As shown in the figure, the core technical feature of the BeeGFS-based metadata query efficiency optimization method of the present invention is the construction of a three-dimensional metadata index network. This network is based on a dynamic hierarchical storage architecture of data access popularity. It realizes intelligent scheduling and index optimization of the three-level storage media through a heat quantification model. At the same time, it designs storage engines, index structures and data flow mechanisms that are adapted to each level, thereby improving the metadata query efficiency from four dimensions: storage media, index design, load balancing and data scheduling.

[0047] 1. Core Architecture of 3D Metadata Indexing Network

[0048] The three-dimensional metadata indexing network is based on "heat-based tiering + engine adaptation + index optimization". It divides metadata storage into three tiers, with each tier employing different storage media, index structures, and optimization strategies to achieve the optimal balance between query efficiency and storage cost.

[0049] Top-level index: PMEMKV hot data index cluster, which uses persistent memory (PMEM) as the storage medium to build a distributed index cluster. It is designed for high-frequency query hot data and aims to achieve microsecond-level query response.

[0050] Intermediate layer index: RocksDB distributed metadata storage uses NVMe SSD as storage medium to build a sharded distributed storage cluster, which is adapted to frequently queried data with medium to high frequency, and aims to achieve millisecond-level query throughput;

[0051] Underlying index: ZFS cold data archiving system, which adopts a hybrid storage architecture of SATA SSD + mechanical hard drive to build an archive storage system that is adapted to cold data with low frequency of queries. The goal is to ensure query availability while controlling storage costs.

[0052] The three levels enable dynamic flow of metadata through a hot data quantification model. Hot data is automatically promoted to higher levels (PMEMKV / RocksDB), while cold data is automatically degraded to lower levels (RocksDB / ZFS). This ensures that high-level storage resources are always used to serve metadata with high query demand. At the same time, through index optimization design at each level, query latency is further reduced and query throughput is improved.

[0053] 2. The heat quantification model enables hierarchical scheduling of metadata.

[0054] To accurately identify hot and cold data and enable intelligent flow of metadata among the three-level indexes, a hotness quantification model based on multi-dimensional features is designed. The metadata hotness value H is calculated through a weighted scoring mechanism and serves as the basis for determining whether data is promoted or downgraded, ensuring that higher-level storage resources prioritize serving metadata with high query demand.

[0055] 2.1 Formula for Calculating Heat Value

[0056] Formula for calculating heat value:

[0057] F represents the access frequency per unit time (times / second), reflecting the density of metadata queries. The weight α = 0.5 (core indicator) because the query frequency directly determines the urgency of metadata's demand for high-level storage resources. High-frequency query data should prioritize high-performance storage.

[0058] T is the interval (in seconds) between the most recent access time and the current time. This reflects the decay characteristic of "the more recent the access time, the higher the popularity", with a weight β=0.3. For example, metadata accessed 10 seconds ago... =0.1, metadata accessed 100 seconds ago =0.01, ensuring that recently queried metadata is retained at a high level, adapting to the temporal locality characteristics of business queries;

[0059] B represents the average bandwidth per access (MB / s), reflecting the resource consumption intensity of metadata queries. The weight γ=0.2. High-bandwidth metadata queries (such as large file metadata) are prone to IO blocking when queried in low-performance storage, and high-performance storage resources should be allocated first.

[0060] 2.2 Judgment Threshold Criteria and Adaptive Adjustment Strategy

[0061] Basic judgment threshold:

[0062] When H≥2.0, it is determined to be hot data, triggering the promotion process from the bottom layer to the middle layer to the top layer, ensuring that the hot data enters the PMEMKV cluster and obtains a microsecond-level query response;

[0063] When H≤0.1, it is determined to be cold data, triggering a degradation process from top level to middle level to bottom level, releasing high-level storage resources to hot data that needs them more.

[0064] The popularity score calculation formula can adaptively adjust weights based on the scenario, supporting dynamic adjustment of weight parameters according to business scenarios to adapt to the query characteristics of different scenarios and further improve the accuracy of stratification. The specific adjustment rules are as follows:

[0065] High-performance computing scenarios (small files account for ≥80%, single file size <100MB): intensive queries of small files and access frequency are the core requirements. Adjust α to 0.6, β=0.2, γ=0.2 to ensure that the metadata of small files with high frequency of queries is prioritized and promoted to the top level.

[0066] Artificial intelligence training scenario (large files account for ≥60%, single file size >1GB, single query bandwidth >100MB / s): large files are loaded in batches and single query bandwidth is high. Adjust γ to 0.3, α=0.5, β=0.2 to ensure that the metadata of high bandwidth queries is supported by high-level storage and avoid query IO blocking.

[0067] In big data archiving scenarios (file access interval > 7 days, monthly query frequency < 1 time): the query frequency is low and the time decay characteristics are obvious. Adjusting β to 0.4, α=0.4, and γ=0.2 can accelerate the identification and degradation of cold data.

[0068] 2.3 Data Flow Mechanism

[0069] The core objective of data flow is to achieve a smooth migration of metadata between three-level indexes without affecting query services, while ensuring the correctness and low latency of query requests during the migration process.

[0070] In addition to meeting the basic judgment threshold, the following conditions must also be met before a promotion or demotion operation can be performed:

[0071] When H ≥ 2.0, the following conditions must be met for thermal data promotion:

[0072] If the H value of the metadata in the underlying ZFS is stable at ≥1.5 for 5 consecutive minutes, the first stage of promotion is triggered: migration to the middle-level RocksDB storage. The migration adopts an incremental synchronization mechanism. During the migration, read locks are added (allowing query requests to continue accessing the original data in ZFS). After the migration is completed, the query route is switched through atomic operations to ensure that the query is unaware of the migration.

[0073] If the H value of the metadata in the middle-layer RocksDB is stable at ≥2.0 for 3 consecutive minutes, the second-stage promotion is triggered: migration to the top-layer PMEMKV cluster. Before migration, the target MDS node (metadata server) is calculated through consistent hashing. The migration process adopts the "synchronize first and then switch" strategy. After synchronization is completed, the index is created on the target node, and then the query route is switched to PMEMKV to avoid query failure during migration.

[0074] When H ≤ 0.1, cold data is degraded if the following conditions are met:

[0075] If the H value in the metadata of the top-level PMEMKV is consistently ≤0.3 for one consecutive hour, the first stage of degradation is triggered: migration to the middle-level RocksDB storage. Before migration, the load of the corresponding shard in the middle layer is checked. If the load exceeds the threshold (CPU utilization ≥70% or IO utilization ≥80%), the migration is delayed until the load decreases. During the migration, the cached copy in PMEMKV is retained until the middle-level query routing takes effect, and then it is deleted to ensure that the query latency does not increase during the degradation period.

[0076] If the H value of the metadata in the middle layer RocksDB is stable at ≤0.1 for 24 consecutive hours, the second stage of degradation is triggered: migration to the underlying ZFS archive system. The migration adopts a batch synchronization mechanism (migrate in batches by item ID), and the migration rate is controlled within 100MB / s to avoid occupying too much bandwidth and affecting normal queries. After the migration is completed, the metadata index pointer (including ZFS storage path and data verification value) is retained in the middle layer to facilitate quick location during subsequent queries.

[0077] 3. Top-level index: PMEMKV hot data index cluster

[0078] The top-level index is designed for hot data (high-frequency queries). Its core goal is to achieve microsecond-level query response and support for high-concurrency queries. Through PMEM storage media, distributed architecture, key-value design and index engine, it solves the problems of high query latency and weak concurrency capabilities of the native solution.

[0079] 3.1 Distributed Architecture Design and Load Balancing

[0080] A distributed architecture of "multiple MDS nodes + consistent hashing" is adopted. Each MDS node (hardware specifications: CPU ≥ 32 cores (Intel Xeon 8375C), memory ≥ 128GB, PMEM ≥ 2TB) deploys an independent PMEMKV instance (persistent memory key-value store). Metadata is distributed to each node through a consistent hashing algorithm to achieve query load balancing. The specific implementation is as follows:

[0081] Consistent hashing algorithm selection: The MurmurHash3-128 hash function is adopted. This function has the characteristics of uniform hash value distribution and fast calculation speed (the hash calculation time for a single piece of metadata is ≤10ns). Compared with MD5 (calculation time ≥50ns) and SHA-1 (calculation time ≥30ns), it can reduce the calculation latency of query routing.

[0082] Hash ring construction: Each MDS node's PMEMKV instance is mapped to multiple virtual nodes on a hash ring (each node maps to 16 virtual nodes), with the hash values ​​of the virtual nodes distributed between 0 and 2. 128 Within the specified range, the metadata key is calculated using MurmurHash3-128 to obtain a hash value. Based on the position of the hash value on the hash ring, the MDS node corresponding to the nearest virtual node in the clockwise direction is selected as the storage node.

[0083] Load balancing advantages: Compared to the traditional "hash modulo" load balancing method, consistent hashing only needs to migrate the metadata corresponding to adjacent virtual nodes on the hash ring when adding or taking offline MDS nodes (migration ratio = 1 / number of nodes), while hash modulo requires the migration of most metadata (migration ratio = (number of nodes - 1) / number of nodes). For example, when the number of MDS nodes expands from 4 to 5, the metadata migration ratio of consistent hashing is only 20%, while the migration ratio of hash modulo is 80%, which greatly reduces the migration overhead and query performance fluctuations during the expansion process.

[0084] Dynamic load balancing: Real-time monitoring of query QPS (queries per second) and PMEM utilization for each MDS node. When a node's QPS exceeds a threshold (default 50,000 queries / second) or its PMEM utilization exceeds 80%, virtual node migration is triggered. The 50,000 queries / second and 80% are the first single-node thresholds for QPS and PMEM utilization, respectively. Virtual nodes for migration are selected based on a "load share priority + popularity filtering" principle: first, the top 30% of virtual nodes contributing the most to the load (based on query volume over the last 5 minutes) are selected; then, virtual nodes with a popularity value H ≥ 1.0 are excluded; finally, the highest-loaded virtual nodes are selected for migration. The target nodes are low-load nodes in the cluster with a QPS below 30,000 queries / second and a PMEM utilization below 60%. The migration process employs an "incremental synchronization + dual-write query" strategy to ensure uninterrupted and unlost queries during migration.

[0085] 3.2 Key-value Design

[0086] The core objective of key-value pair design is to shorten key matching time during queries, reduce metadata storage footprint, improve PMEM storage density, and ensure key uniqueness. The specific design is as follows:

[0087] Key structure design: A five-tuple structure is adopted: {item UUID}{directory inode}{filename hash}{access feature vector hash}{version number}, with the total length controlled within 48 bytes. The function and design advantages of each field are as follows:

[0088] Project UUID (16 bytes): Distinguishes metadata for different business projects and avoids cross-project metadata conflicts;

[0089] Directory inode (8 bytes): Locates the directory where metadata is located, adapting to directory-level query scenarios (such as directory traversal);

[0090] Filename hash (16 bytes): The filename is hashed using the SHA-1 algorithm and then truncated to 16 bytes. Compared with storing the complete filename (which can be up to 255 bytes), this greatly shortens the key length and reduces hash calculation and matching time.

[0091] Access Feature Vector Hash (4 bytes): A hash value generated based on the access frequency and bandwidth characteristics of metadata, used to quickly filter metadata that matches the query characteristics;

[0092] Version number (4 bytes): Records the updated version of metadata to avoid querying outdated data.

[0093] Design advantages: Compared to the native BeeGFS "inode + filename" key structure (which can be up to 263 bytes long), the key length of this design is shortened by 81.7%. When stored in PMEM, the same capacity of PMEM can store more metadata (storage density is increased by 2.6 times). The memory bandwidth usage for key matching during queries is reduced, and the matching speed can be increased by more than 3 times. At the same time, the five-tuple structure ensures the uniqueness of keys in scenarios with multiple projects, multiple directories, and files with the same name, avoiding the return of incorrect data during queries.

[0094] Value structure design: It adopts Protocol Buffers (Lightweight Binary Data Interchange Format) for serialization and storage, which includes 12 core metadata fields (inode ID, access timestamp, lock status, data block distribution information, file size, creation time, modification time, permission information, owner ID, group ID, file type and check value), and the serialized size is controlled within 64 bytes.

[0095] Design advantages: Compared to JSON (serialized size ≥ 128 bytes) and XML (serialized size ≥ 256 bytes), Protocol Buffers reduces the size of serialized data by 50%-75%, and the deserialization time during queries is ≤ 50ns, which is more than 4 times faster than the deserialization time of JSON (≥ 200ns), further reducing query latency.

[0096] 3.3 Core Engine Optimization

[0097] Deep optimizations were performed on the PMEMKV storage engine to address issues such as lock contention, invalid queries, and memory fragmentation under high-concurrency queries. Specific optimization measures are as follows:

[0098] Multi-way parallel hash bucket structure: The hash table is divided into 1024 independent hash buckets, each hash bucket corresponding to an independent read-write lock (shared read lock, exclusive write lock). Compared with the traditional global lock design, the lock contention rate during concurrent queries is reduced by more than 99%. For example, in a high-concurrency query scenario of 100,000 QPS, the lock wait time of the global lock design accounts for 30%, while the lock wait time of the multi-way parallel hash bucket structure is ≤0.5%.

[0099] Pre-filtering with Bloom filter: Deploy a Bloom filter (false positive rate ≤ 0.01%) in front of each hash bucket. When querying, first check if the metadata exists in the hash bucket using the Bloom filter. If it does not exist, return "miss" directly. There is no need to traverse the linked list nodes in the hash bucket, which reduces lock contention and traversal overhead for invalid queries. In the case of metadata miss, the query latency is reduced by more than 60%.

[0100] Fine-grained memory allocation: The Jemalloc memory allocation tool is used to pre-allocate fixed-size memory blocks based on the size of the metadata value (64 bytes), avoiding memory fragmentation caused by frequent memory allocation and deallocation. The memory fragmentation rate is controlled within 5%. Compared with the default memory allocation tool (memory fragmentation rate ≥20%), PMEM improves resource utilization by more than 15% and reduces the impact of memory fragmentation on query performance.

[0101] 3.4 Heat Management and Sustainability

[0102] Dual-threshold dynamic elimination mechanism:

[0103] Single node threshold: When the PMEM utilization rate of a certain MDS node reaches 85%, where 85% is the second single node threshold for PMEM utilization, the single node LRU (Least Recently Used) eviction is initiated, prioritizing the eviction of metadata with lower H values ​​(H≤0.5) to release PMEM space.

[0104] Global Threshold for the Cluster: When the average PMEM utilization of the entire PMEMKV cluster reaches 80% (global threshold), the low-hot metadata of some nodes will be downgraded to the mid-level RocksDB. This mechanism is a different independent strategy from the migration of single nodes with PMEM utilization exceeding 80% in the dynamic load adjustment in section 3.1 (this applies to the overall cluster load, while section 3.1 applies to single node overload). The selection rule is as follows: prioritize the selection of nodes in the cluster with PMEM utilization ≥ 85% and low-hot metadata (H ≤ 0.3) accounting for ≥ 20%, then sort the nodes by the amount of low-hot metadata in descending order, and select the top N nodes (N is 30% of the total number of nodes in the cluster, and no more than 5). During the demotion, only the low-hot metadata with H ≤ 0.3 on these nodes is migrated to the mid-level RocksDB to ensure that the average PMEM utilization of the entire cluster drops below 70% and to ensure stable cluster query performance.

[0105] Popularity prediction and pre-reservation: Based on the query frequency and historical cycle pattern of the past hour (such as the daily peak query period from 9 to 12 o'clock), the metadata that may become hot data in the next 10 minutes is predicted by the sliding window algorithm, and PMEM space is reserved in advance. When this metadata is frequently queried, it is directly loaded into the reserved space to avoid the loss of hot data due to LRU eviction, and the query hit rate is improved by more than 20%.

[0106] Persistence Guarantee: Employing an "asynchronous WAL (Write-Ahead Log) + incremental Checkpoint" mechanism ensures that metadata is not lost and recovery is fast.

[0107] Asynchronous WAL: When metadata is updated, it is first written to the WAL log (stored in the log area of ​​PMEM), and then the hash table is updated. The WAL log is written in batch (a batch write is triggered every 1,000 updates), and the log write latency is controlled to be ≤1 microsecond.

[0108] Incremental Checkpoint: An incremental snapshot is triggered every 30 minutes, backing up only the metadata that has changed since the last snapshot. The snapshot is stored in the snapshot area of ​​PMEM. During recovery, only the latest snapshot and subsequent WAL logs need to be loaded, and the recovery time is ≤30 seconds. Compared with full snapshot (recovery time ≥5 minutes), it greatly improves the query availability after fault recovery.

[0109] 4. Intermediate-tier index: RocksDB distributed metadata storage

[0110] The intermediate-level index is designed for frequently queried data with medium to high frequency. Its core goal is to achieve high query throughput and support complex conditional queries. At the same time, it serves as a buffer between the top and bottom layers, facilitating the degradation of hot data and the promotion of cold data. The specific implementation is as follows:

[0111] 4.1 Sharded Distributed Architecture and Consistency Guarantee

[0112] The system uses a composite partition key of "directory inode + project ID" to achieve horizontal sharding storage of metadata. Each shard corresponds to an independent RocksDB instance, deployed on NVMe SSD nodes. The specific architecture is as follows:

[0113] Sharding rules: The directory inode (8 bytes) and project ID (8 bytes) are concatenated and then hashed (using CRC32 hash function). Based on the hash value, the metadata is allocated to 64 shards (supports dynamic expansion to 256). Each shard is deployed with 3 replicas (primary replica + 2 secondary replicas). Replica consistency is achieved through the Raft protocol.

[0114] Raft protocol optimization: The election timeout of the Raft protocol has been adjusted to 500ms (default 1000ms). Log replication adopts a batch synchronization mechanism (batch replication is triggered every 500 logs). The replica synchronization latency is controlled within 10ms to ensure that when the primary replica fails, the secondary replica can quickly switch to the primary replica. The query interruption time is ≤1 second.

[0115] Complex query support: Multi-dimensional inverted indexes can be built using column family design to meet query requirements with different conditions.

[0116] Main column family (cf_main): Stores the mapping from filenames to metadata, supporting precise queries by filename;

[0117] Auxiliary column family cf_ctime: Stores the mapping from creation time to metadata, supporting queries by creation time range;

[0118] Auxiliary column family cf_size: Stores the mapping from file size to metadata, supporting queries by file size range;

[0119] The auxiliary column family cf_access stores the mapping between access characteristics and metadata, and supports querying by access frequency and bandwidth characteristics.

[0120] Design advantages: Compared to the single index structure of native BeeGFS (which only supports queries by inode), this design supports multi-dimensional condition combination queries, such as "querying the metadata of files created after December 1, 2025, in project A with a size between 100MB and 1GB", without having to traverse the entire metadata, improving query efficiency by more than 10 times.

[0121] 4.2 Storage Engine Optimization and Cache Coordination

[0122] Parameter tuning was performed on RocksDB's LSM tree (Log Structure Merge Tree), and a two-level caching architecture was designed to improve query throughput and hit rate.

[0123] LSM tree parameter tuning:

[0124] The maximum number of files in the L0 layer is set to 8 (4 by default), which reduces the number of merges between the L0 and L1 layers and lowers the CPU usage (the CPU usage for merge operations is reduced from 60% to 30%).

[0125] The size multiplier for L1-L6 layers is set to 4 (default 10), reducing the number of layers (from 7 to 5), lowering the read amplification factor (from 10x to 5x), and reducing the number of disk I / O operations per query by 50%.

[0126] With the Bloom filter bit width set to 10 for each data block (default 8), the false positive rate is reduced from 0.05% to 0.01%, reducing unnecessary disk I / O.

[0127] Two-level cache collaboration:

[0128] Node Local Block Cache: Allocate 20% of the node's memory as a local block cache. This means that 20% of the total physical memory of a single server node (physical server or virtualized server) deploying a RocksDB instance is used as a local block cache. This cache is specifically used to cache RocksDB's SST file data blocks, with a cache hit rate of ≥70%, reducing disk I / O.

[0129] Distributed Redis caching: Deploy an independent Redis cluster to cache metadata that was hit by queries within the last hour. Set the cache expiration time to 1 hour. When querying across nodes, prioritize accessing the Redis cache. The hit rate is ≥60%, avoiding the network overhead of accessing RocksDB across nodes.

[0130] Prefetch strategy: Based on directory access correlation, when querying the metadata of a parent directory, the metadata of the first 10 subdirectories under that directory is automatically prefetched into the local block cache. For example, in the directory traversal scenario of weather simulation, the prefetch hit rate is ≥80%, and the directory traversal efficiency is improved by more than 3 times.

[0131] 4.3 Hierarchical Adaptive Compression Strategy

[0132] Different compression algorithms are used for SST files at different levels to achieve a balance between storage efficiency and decompression speed. The specific strategy is as follows:

[0133] L0-L2 layers: The LZ4 compression algorithm is used (decompression speed ≥ 400MB / s). Files in this layer are accessed frequently, so decompression speed is prioritized. The compression ratio is approximately 2:1.

[0134] Layers L3-L6: The ZSTD compression algorithm is used (compression ratio of approximately 3:1). Files in this layer are accessed less frequently, so storage efficiency is prioritized and decompression speed is ≥200MB / s.

[0135] Design advantages: Compared to a single compression algorithm (such as using only LZ4 or ZSTD), the hierarchical adaptive compression strategy reduces metadata storage space by more than 30%, while keeping decompression latency within 1 microsecond, without affecting query response speed.

[0136] 5. Underlying index: ZFS cold data archiving system (implemented with low-cost, high-availability queries)

[0137] The underlying index is designed for cold data with low-frequency queries. Its core goal is to ensure query availability and data security while controlling storage costs, and to support quick retrieval to the intermediate index. The specific implementation is as follows:

[0138] 5.1 “Storage Pool + Dataset” Architecture Design

[0139] A hierarchical archiving system is built based on ZFS (Zettabyte File System), adopting a "storage pool + dataset" architecture. Each dataset corresponds to cold data with a specific access frequency. The specific design is as follows:

[0140] Storage pool configuration: Each storage pool consists of 16 NVMe SSDs (cache layer) and 64 mechanical hard drives (capacity layer), and enables ZFS's enhanced adaptive compression strategy. It intelligently matches LZ4 or GZIP compression algorithms through metadata features to achieve a dynamic balance between storage efficiency and access performance. The average compression ratio reaches 3:1, and the IO response speed during queries is improved by more than 2 times compared to pure mechanical hard drive storage.

[0141] Algorithm selection logic: For metadata types with high frequency of access (≥3 accesses in the last hour) and low redundancy (repeated byte sequence ratio <20%), such as file inode basic information (inode ID, permission configuration, timestamp, data block distribution pointer) and directory entry index, the LZ4 compression algorithm is automatically selected. This algorithm has a decompression speed of up to 2GB / s, and its CPU overhead is only 1 / 5 of GZIP. It can achieve a compression ratio of 1.5:1 to 2:1 with almost no increase in query latency. For metadata types with low frequency of access (<1 access in the last 7 days) and high redundancy (repeated byte sequence ratio ≥20%), such as file extended attributes (user-defined descriptions, batch file unified comments), metadata snapshot logs, and historical access records, the GZIP-4 level compression algorithm (medium compression level) is automatically selected. Its compression ratio can reach 2.5:1 to 5:1, which can significantly reduce storage usage. Moreover, due to the low access frequency, the decompression latency (≤1ms) has no significant impact on the overall query performance.

[0142] Dynamic adjustment mechanism: The ZFS background process evaluates the performance and efficiency of compressed metadata every hour. If the decompression latency of a certain type of metadata after using GZIP exceeds the preset threshold (>1ms) or the actual compression ratio is lower than 1.8:1 (not achieving the expected storage benefits), the background process will automatically trigger recompression to LZ4. If the storage usage of LZ4 compressed metadata is too high (compression ratio <1.5:1) and the access frequency continues to decline (access <1 time in the last 3 days), the metadata will be asynchronously recompressed to GZIP-6 level (higher compression ratio) to ensure that the overall compression strategy always adapts to the changes in the access characteristics of metadata.

[0143] Dataset partitioning: Based on the access frequency of cold data, the datasets were divided into three datasets (ds_cold1, ds_cold2, ds_cold3):

[0144] ds_cold1: H value is between 0.05 and 0.1, queried 1-5 times per month, stored in the NVMe SSD cache layer, with a query latency of ≤5ms.

[0145] ds_cold2: H value between 0.01 and 0.05, queried 1-3 times per quarter, stored in a high-speed partition of the mechanical hard disk capacity layer, with a query latency of ≤20ms.

[0146] ds_cold3: H value < 0.01, queried less than once a year, stored in the archive partition of the mechanical hard disk capacity tier, query latency ≤ 50ms.

[0147] Snapshots and Recovery: ZFS snapshot mechanism is enabled, creating a full snapshot weekly and an incremental snapshot daily. It supports restoring metadata at specific points in time, meeting data compliance requirements. The snapshot creation process does not affect query performance (IO utilization ≤ 10%).

[0148] 5.2 Secondary Index Design and Query Optimization

[0149] To address the inefficiency of traversing the entire dataset when querying cold data, a secondary index for SQLite is constructed, with the following design:

[0150] Primary index: It uses the mapping relationship of inode → storage path and is stored in the main table of SQLite. The primary table index adopts a B+ tree structure and supports fast query of the storage path of metadata by inode, with a query time of ≤1ms.

[0151] Secondary index: It adopts the mapping relationship of {project ID, creation time} → inode list and is stored in the secondary table of SQLite. The secondary table index also adopts B+ tree structure, which supports batch querying of inode list by project ID and creation time range, and then querying specific metadata through the primary index. For example, "query all cold data metadata created in 2023 in project B". The query time is ≤100ms, which is more than 100 times more efficient than the traversal query without index (time ≥10 seconds).

[0152] Read-ahead caching acceleration: When cold data is queried, the metadata and 10 adjacent metadata records are automatically loaded into ZFS's read-ahead cache (capacity of 10GB). If there are subsequent related queries, they can be read directly from the cache, reducing query latency by more than 60%.

[0153] 5.3 Cold Data Recall Mechanism

[0154] When cold data is queried (H value temporarily rises to ≥0.5), the cold data retrieval mechanism is triggered to quickly migrate it to the middle-layer RocksDB. The specific process is as follows:

[0155] Recall Trigger: After a query hits the underlying ZFS metadata, the system automatically records the number of queries. If the number of queries is ≥3 within 24 hours, the recall process is triggered.

[0156] Incremental synchronous migration: An incremental synchronous mechanism is adopted to migrate only the core fields of metadata (consistent with the metadata structure of the middle layer RocksDB), the migration rate is controlled at 50MB / s to avoid consuming too much bandwidth, and the migration time is ≤1 second (single metadata).

[0157] Query route switching: After the migration is completed, the query route for this metadata is switched to the middle layer RocksDB through atomic operations. Subsequent queries will directly access the middle layer, reducing query latency from tens of milliseconds to milliseconds.

[0158] To better illustrate this method, a specific example will be used below for further explanation:

[0159] The complete process of metadata query is as follows:

[0160] After metadata is generated / integrated into the system, it is allocated to the middle-layer RocksDB storage based on the initial access characteristics (default H=0.5). If the initial access frequency is ≥10 times / second (H≥2.0), it is directly allocated to the top-layer PMEMKV cluster.

[0161] When a metadata query request is received, the query process is executed in the following order: "Top-level PMEMKV → Middle-level RocksDB → Bottom-level ZFS".

[0162] If the metadata requested in the query is hot data (already in the top-level PMEMKV): find the corresponding MDS node through consistent hashing, query the hash table of PMEMKV, and if a match is found, directly return the serialized metadata value, deserialize it and provide it to the business system. The query latency is ≤50 microseconds.

[0163] If the metadata of the query request is frequently used data (already in the middle layer RocksDB): find the corresponding shard through the composite partition key, query the column family index of RocksDB, and return the result if a match is found, with a query latency of ≤5 milliseconds; if the local block cache is not hit, load it from the disk and synchronize it to the cache;

[0164] If the metadata requested in the query is cold data (already in the underlying ZFS): find the storage path through the SQLite secondary index, read the metadata from the ZFS storage pool, and return the result with a query latency of ≤50 milliseconds; if the recall conditions are met, the migration to the middle layer RocksDB will be triggered.

[0165] The system monitors query metrics (QPS, response latency, and hit rate, etc.) at each level in real time. When the QPS of a node in the top-level PMEMKV reaches ≥50,000 times / second, dynamic load adjustment is initiated. When there is unbalanced shard load in the middle-level RocksDB (QPS difference between shards ≥30% or CPU / IO utilization difference ≥25%), shard migration is triggered (that is, migrating some metadata from high-load shards to low-load shards, using an incremental synchronization + dual-write strategy to ensure uninterrupted queries). When the average heat value H of the metadata of a dataset in the bottom-level ZFS reaches ≥0.08, the dataset storage configuration is adjusted (migrating this part of the metadata from the mechanical hard disk archive partition to the NVMe SSD cache layer or the mechanical hard disk high-speed partition) to maintain the consistency between the storage configuration and the metadata heat value.

[0166] The heat quantification model calculates and updates the H value of metadata in real time based on data collected from real-time monitoring (calculated every 10 seconds), and triggers promotion or demotion processes based on the H value:

[0167] Hot data promotion: bottom layer ZFS → middle layer RocksDB → top layer PMEMKV, executed according to the flow mechanism in section 2.3;

[0168] Cold data degradation: top layer PMEMKV → middle layer RocksDB → bottom layer ZFS, executed according to the flow mechanism in section 2.3.

[0169] This method, through a three-level index architecture and hierarchical storage optimization design, has been experimentally verified (experimental environment: using a 4-node PMEMKV cluster + an 8-node RocksDB cluster + a 16-node ZFS cluster; the test dataset contains 50 million metadata entries, covering small files, large files, and archive file scenarios). The query latency for hot data has been reduced from 320-450 microseconds in native BeeGFS to 35-58 microseconds (a reduction of 7.8-12.9 times, or an order of magnitude), the query latency for frequently used data has been reduced from 35-62 milliseconds to 3.2-5.8 milliseconds (a reduction of 6.4-18.1 times), and the query latency for cold data has been reduced from 1.2-3.5 seconds to 35-48 milliseconds (a reduction of 25-100 times), meeting the query latency requirements.

[0170] Performance tests show that the top-level PMEMKV cluster achieves a single-node concurrent query QPS of 52,000-58,000 (default threshold of 50,000 queries / second), and a 4-node cluster-level concurrent throughput of 250,000-280,000 QPS, representing a performance improvement of 4.2-4.8 times compared to the native BeeGFS single-node QPS limit of 10,000. The mid-level RocksDB cluster supports 21,000-23,000 QPS per shard, and the total concurrent throughput of 32 shards reaches 670,000-730,000 QPS, meeting the needs of high-concurrency query scenarios.

[0171] In complex query scenarios, the mid-level RocksDB cluster, through multi-dimensional reverse indexes and prefetching strategies, reduces the time spent traversing multi-level nested directories (100,000-level files) from the original 8.2-12.5 seconds to 2.1-3.8 seconds (an efficiency improvement of 3.3-5.9 times). The time spent on multi-condition combined queries (such as filtering by project ID + creation time + file size) has been reduced from 15-28 seconds to 1.2-2.5 seconds (an efficiency improvement of 10-23.3 times), completely solving the problem of inefficient queries in scenarios with large file sets.

[0172] In terms of resource utilization, the dynamic tiered storage mechanism increases the utilization of high-performance storage resources such as PMEM and NVMe SSDs from 32%-38% in the native solution to 72%-79%; after cold data is migrated to low-cost mechanical hard drives, the overall storage cost is reduced by 42%-48% compared to the full high-performance storage solution, achieving the optimal balance between performance and cost.

[0173] The BeeGFS-based metadata query efficiency improvement system of the present invention includes:

[0174] Network building module: used to build a three-dimensional metadata index network, specifically including a top-level index using the PMEMKV hot data index cluster, an intermediate-level index using the RocksDB distributed metadata storage, and a bottom-level index using the ZFS cold data archiving system;

[0175] Metadata classification module: This module is used to divide all metadata into three independent sets based on popularity: hot data, frequently used data, and cold data, and store them in the top-level index, intermediate-level index, and bottom-level index respectively.

[0176] The query module is used to query the top-level index, intermediate index, and bottom-level index in sequence after receiving a query request. If a match is found, the corresponding metadata is returned; otherwise, an empty value is returned directly.

[0177] Monitoring module: Used to monitor query metrics at each level in real time, update metadata popularity values ​​and load status at each level in real time; adjust storage configuration at each level based on the latest popularity values ​​and load status at each level.

[0178] Data migration module: used to dynamically promote or downgrade metadata across the 3D metadata index network based on the latest popularity value and preset popularity threshold. The promotion or downgrade process first synchronizes the data and then switches the query path.

Claims

1. A method for improving metadata query efficiency based on BeeGFS, characterized in that, Includes the following processes: A three-dimensional metadata index network is constructed, which specifically includes a top-level index using the PMEMKV hot data index cluster, an intermediate-level index using the RocksDB distributed metadata storage, and a bottom-level index using the ZFS cold data archiving system. All metadata is divided into three independent sets according to popularity value: hot data, frequently used data, and cold data, and stored in the top-level index, intermediate-level index, and bottom-level index respectively. Upon receiving a query request, the system queries the top-level index, intermediate-level index, and bottom-level index in sequence. If a match is found, the corresponding metadata is returned; otherwise, a null value is returned directly. Real-time monitoring of query metrics at each level is used to update metadata popularity values ​​and load status at each level in real time. Adjust the storage configuration of each tier based on the latest popularity value and load status of each tier; Based on the latest popularity value and preset popularity threshold, metadata is dynamically promoted or downgraded across the three-dimensional metadata index network. The promotion or downgrade process involves synchronizing data first and then switching query paths.

2. The method according to claim 1, characterized in that, The specific architecture of each layer of the three-dimensional metadata indexing network includes: Top-level index: A distributed index cluster is built using persistent memory as the storage medium and PMEMKV as the storage engine; Intermediate tier index: NVMe-SSDs are used as the storage medium and RocksDB as the storage engine to build a sharded distributed storage cluster; Underlying index: An archive storage system is built using SATA-SSD and mechanical hard drive as storage media and ZFS as the file system.

3. The method according to claim 1, characterized in that: The formula for calculating the heat value is as follows: ， Where H is the heat value, , and These are the weighting coefficients. Here, T represents the metadata access frequency, T represents the interval between the most recent access time of the metadata and the current time, and B represents the average bandwidth of a single metadata access.

4. The method according to claim 3, characterized in that: The promotion process is triggered when H ≥ 2.0, and the demotion process is triggered when H ≤ 0.

1. After the promotion process is triggered, further evaluation is required. Promotion can only proceed if the following conditions are met: When metadata in the underlying index is promoted to the intermediate index, H must be ≥ 1.5 for 5 consecutive minutes. When metadata from an intermediate-level index is promoted to the top-level index, it must meet the condition that H ≥ 2.0 for 3 consecutive minutes. After the downgrade process is triggered, further judgment is required. The downgrade operation can only proceed if the following conditions are met: When metadata in a top-level index is downgraded to an intermediate-level index, H must be less than or equal to 0.3 for one consecutive hour. When intermediate-level index metadata is downgraded to the underlying index, H must be less than or equal to 0.1 for 24 consecutive hours.

5. The method according to claim 2, characterized in that: The top-level index uses the MurmurHash3-128 hash function to map each MDS node to a hash ring. The number of virtual nodes on the hash ring = 16 × the number of physical MDS nodes. The key of the top-level index adopts a 5-tuple structure, which includes the item UUID, directory inode, filename hash, access feature vector hash, and version number. The total length of the 5-tuple is ≤48 bytes. Among them, the 16 bytes truncated after hashing the filename using the SHA-1 algorithm are used as the filename hash. The value is serialized using Protocol Buffers and includes 12 core metadata fields with a length ≤64 bytes. The PMEMKV storage engine internally divides the hash table into multiple independent hash buckets, each hash bucket corresponding to an independent read-write lock. The read lock is shared and the write lock is exclusive. Each hash bucket has a Bloom filter in front. When querying, if the Bloom filter determines that the key does not exist, it immediately returns a miss. If the key exists, it enters the corresponding bucket for key matching.

6. The method according to claim 5, characterized in that: The top-level index monitors the query QPS and PMEM utilization of each MDS node in real time. When the QPS or PMEM utilization of any node exceeds its first single-point threshold, the metadata of some virtual nodes of that node is migrated to low-load virtual nodes. The migration adopts an incremental synchronization and query dual-write strategy. When the PMEM utilization of a single node exceeds the second single-point threshold, single-node LRU eviction is enabled. The second single-point threshold of PMEM utilization is greater than its first single-point threshold. When the average PMEM utilization of the cluster exceeds the global threshold, metadata with a heat value lower than the preset value is downgraded to the intermediate index. The top-level index predicts future hot data and reserves PMEM space for it based on the query frequency and historical cycle patterns within a preset time period using a sliding window algorithm. When metadata is updated, the top-level index first writes the metadata to the WAL log and then updates the hash table, with the WAL log being written in batch processing. The top-level index is periodically snapshotted incrementally, backing up only the metadata that has changed since the last snapshot and storing it in the snapshot area of ​​PMEM.

7. The method according to claim 2, characterized in that: The intermediate layer index concatenates the directory inode and project ID of the metadata and performs hash calculation. Based on the hash value, the metadata is sharded and stored. Each shard is deployed with multiple replicas and the Raft protocol is used to ensure replica consistency. The intermediate tier index has multiple column families that map different types of metadata to support multidimensional condition combination queries; The intermediate tier index increases the maximum number of files in the L0 layer of RocksDB's LSM, reduces the size multiplication factor of the L1-L6 layers, and increases the number of bits in the Bloom filter for each data block; The intermediate index layer uses 20% of the total physical memory of a single server node as a local block cache and deploys an independent Redis cluster to cache the metadata hit by queries within a preset time. A cache expiration time is set for each Redis cluster, and the Redis cache is accessed first when querying across nodes. When querying metadata for a parent directory, the metadata of the first 10 subdirectories under that directory is automatically prefetched into the local block cache; RocksDB's LSM uses different compression algorithms for different levels of files in the intermediate index layer, with higher-level compression algorithms achieving lower compression ratios than lower-level algorithms.

8. The method according to claim 2, characterized in that: The underlying index matches different compression algorithms based on metadata characteristics according to ZFS's enhanced adaptive compression strategy, and periodically evaluates the performance and efficiency of the compressed metadata. For metadata whose decompression latency, compression ratio, storage usage, and access frequency reach preset standards, a new compression algorithm is selected for recompression. The underlying index determines its storage location based on the popularity value of metadata and the number of queries per month; The underlying index is used to build SQLite secondary indexes, specifically including: Primary index: It uses the inode→storage path mapping relationship and is stored in the main table of SQLite; Secondary indexes: Using the mapping relationship of {project ID, creation time} → inode list, they are stored in a secondary table of SQLite; When cold data is queried for the first time, the original data and several adjacent data entries are automatically loaded into the ZFS read-ahead cache for subsequent related queries.

9. The method according to claim 2, characterized in that: When cold data is queried no less than a preset number of times within a preset time and its popularity value is greater than a preset value in the underlying index, a recall process is triggered: only the core fields of the metadata are migrated to the intermediate index, and after the migration is completed, the query route of the metadata is switched to the intermediate index through atomic operations.

10. A system for improving metadata query efficiency based on BeeGFS, characterized in that, include: Network building module: used to build a three-dimensional metadata index network, specifically including a top-level index using the PMEMKV hot data index cluster, an intermediate-level index using the RocksDB distributed metadata storage, and a bottom-level index using the ZFS cold data archiving system; Metadata classification module: This module is used to divide all metadata into three independent sets based on popularity: hot data, frequently used data, and cold data, and store them in the top-level index, intermediate-level index, and bottom-level index respectively. The query module is used to query the top-level index, intermediate index, and bottom-level index in sequence after receiving a query request. If a match is found, the corresponding metadata is returned; otherwise, an empty value is returned directly. Monitoring module: Used for real-time monitoring of query metrics at each level, and for real-time updating of metadata popularity values ​​and load status at each level; Adjust the storage configuration of each tier based on the latest popularity value and load status of each tier; Data migration module: used to dynamically promote or downgrade metadata across the 3D metadata index network based on the latest popularity value and preset popularity threshold. The promotion or downgrade process first synchronizes the data and then switches the query path.

Images