Metadata hierarchical storage method, electronic device, storage medium, and program product

Abstract

The application discloses a metadata hierarchical storage method, an electronic device, a storage medium and a program product, relates to the technical field of computers, and constructs a three-level metadata hierarchical storage architecture composed of a memory layer, a high-performance storage layer and a large-capacity storage layer, and adopts a directory shard group mode to manage distributed metadata in the high-performance storage layer and the large-capacity storage layer, so that organization management based on directory shards and intelligent hierarchical storage according to directory paths or depths are realized. By migrating cold metadata to a low-cost large-capacity storage layer, the storage cost is significantly reduced; hot metadata is retained in the high-performance layer, so that the access performance is not affected; the directory shard group supports linear expansion of metadata, and meets large-scale management requirements; and differentiated storage improves the utilization rate of storage resources, and avoids performance waste.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for hierarchical storage of metadata, electronic devices, storage media, and program products. Background Technology

[0002] In large-scale storage systems, efficient metadata management is crucial for ensuring overall system performance. With the explosive growth of data volume, the amount of metadata has also increased dramatically, especially in scenarios with trillions of files, posing a significant challenge to metadata management. Currently, metadata is typically stored in a single storage medium, such as an all-flash storage pool built with a three-replica architecture. However, this approach has significant limitations in scenarios with massive amounts of metadata. Existing metadata storage solutions suffer from the following problems: High cost: Trillions of metadata require petabytes of all-flash storage capacity, and the high cost of all-flash storage media leads to a surge in overall storage costs; Performance waste: Infrequently accessed "cold metadata" shares high-performance storage resources with frequently accessed "hot metadata," resulting in underutilization of high-performance storage resources and resource waste; Limited scalability: A single metadata storage pool struggles to cope with the continuous growth of metadata volume, easily becoming a system performance bottleneck and limiting system scalability. Summary of the Invention

[0003] This application provides a metadata hierarchical storage method, electronic device, storage medium, and program product to at least solve the problems of high cost, performance waste, and limited scalability in existing metadata storage solutions in related technologies.

[0004] This application provides a method for hierarchical storage of metadata, including:

[0005] Construct a three-tier metadata storage architecture consisting of a memory layer, a high-performance storage layer, and a large-capacity storage layer;

[0006] In the high-performance storage layer and the large-capacity storage layer, a directory shard group is set up to manage the metadata of the distributed storage. The metadata under the same directory is divided into multiple sub-shards. The sub-item index table is established to realize the association of the sub-shards through the directory-based shard group.

[0007] In response to storing the sub-segment, the directory path or directory depth of the sub-segment is obtained, and the sub-segment is stored in the high-performance storage layer or the large-capacity storage layer according to the directory path or directory depth of the sub-segment.

[0008] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described metadata hierarchical storage methods when executing the computer program.

[0009] Construct a three-tier metadata storage architecture consisting of a memory layer, a high-performance storage layer, and a large-capacity storage layer;

[0010] In the high-performance storage layer and the large-capacity storage layer, a directory shard group is set up to manage the metadata of the distributed storage. The metadata under the same directory is divided into multiple sub-shards. The sub-item index table is established to realize the association of the sub-shards through the directory-based shard group.

[0011] In response to storing the sub-segment, the directory path or directory depth of the sub-segment is obtained, and the sub-segment is stored in the high-performance storage layer or the large-capacity storage layer according to the directory path or directory depth of the sub-segment.

[0012] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described metadata hierarchical storage methods:

[0013] Construct a three-tier metadata storage architecture consisting of a memory layer, a high-performance storage layer, and a large-capacity storage layer;

[0014] In the high-performance storage layer and the large-capacity storage layer, a directory shard group is set up to manage the metadata of the distributed storage. The metadata under the same directory is divided into multiple sub-shards. The sub-item index table is established to realize the association of the sub-shards through the directory-based shard group.

[0015] In response to storing the sub-segment, the directory path or directory depth of the sub-segment is obtained, and the sub-segment is stored in the high-performance storage layer or the large-capacity storage layer according to the directory path or directory depth of the sub-segment.

[0016] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described metadata hierarchical storage methods:

[0017] Construct a three-tier metadata storage architecture consisting of a memory layer, a high-performance storage layer, and a large-capacity storage layer;

[0018] In the high-performance storage layer and the large-capacity storage layer, a directory shard group is set up to manage the metadata of the distributed storage. The metadata under the same directory is divided into multiple sub-shards. The sub-item index table is established to realize the association of the sub-shards through the directory-based shard group.

[0019] In response to storing the sub-segment, the directory path or directory depth of the sub-segment is obtained, and the sub-segment is stored in the high-performance storage layer or the large-capacity storage layer according to the directory path or directory depth of the sub-segment.

[0020] This application constructs a three-tiered metadata hierarchical storage architecture consisting of a memory layer, a high-performance storage layer, and a large-capacity storage layer. Distributed metadata is managed using directory sharding groups in the high-performance and large-capacity storage layers, achieving directory-sharded organization and intelligent hierarchical storage based on directory paths or depth. By migrating cold metadata to the low-cost large-capacity storage layer, storage costs are significantly reduced; hot metadata is retained in the high-performance layer, ensuring unaffected access performance; directory sharding groups support linear expansion of metadata, meeting large-scale management needs; and differentiated storage improves storage resource utilization and avoids performance waste. By constructing a multi-tiered storage architecture combined with a dynamic migration strategy, differentiated storage and efficient management of metadata are achieved. While ensuring the access performance of hot metadata, the storage cost of cold metadata is reduced. Hierarchical storage management, which dynamically allocates storage resources according to metadata access characteristics, achieves a balance between metadata storage costs and access performance, improving system scalability and resource utilization. Attached Figure Description

[0021] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a diagram illustrating the application environment of a metadata hierarchical storage method in one embodiment of this application.

[0023] Figure 2 This is a schematic diagram of a three-level metadata storage architecture in one embodiment of this application;

[0024] Figure 3 This is a schematic diagram of the metadata storage structure in one embodiment of this application;

[0025] Figure 4 This is a schematic diagram illustrating the change in metadata storage structure after adopting hierarchical storage in one embodiment of this application;

[0026] Figure 5 This is a flowchart illustrating a metadata hierarchical storage method in one embodiment of this application;

[0027] Figure 6 This is a structural block diagram of a metadata hierarchical storage device in one embodiment of this application;

[0028] Figure 7 This is an internal structural diagram of a computer device in one embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0030] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0031] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] The metadata hierarchical storage method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 and server 104 communicate via a network. Terminal 102 can access data from server 104 to add corresponding metadata to server 104. Terminal 102 can initiate access and modification operations on metadata from server 104. Server 104 performs distributed storage and migration management of metadata. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Server 104 can be a standalone server or a server cluster consisting of multiple servers.

[0033] like Figure 2 As shown, a three-tiered metadata storage architecture—a memory layer, a high-performance storage layer, and a large-capacity storage layer—can be constructed in server 104 to achieve tiered metadata storage. The memory layer uses DRAM from cluster nodes as a metadata read cache, caching recently accessed hot metadata. The cache invalidation strategy is the Least Recently Used (LRU) algorithm. The high-performance storage layer consists of all-flash storage media and is used to store metadata logs (metadata change operations are first stored on disk using the "metadata log" method; storing the metadata log in the high-performance layer ensures the performance of metadata modifications) and metadata with high read / write performance requirements. The large-capacity storage layer consists of hybrid flash or HDD storage media and is used to store infrequently accessed, cold metadata with low read performance requirements.

[0034] like Figure 3 As shown, a reasonable metadata storage structure is set up, including directory shard groups and sub-item index tables, to achieve efficient management and fast location of metadata.

[0035] After adopting hierarchical storage, the metadata storage structure changes as follows: Figure 4 As shown, the metadata is split into two parts: file attribute information and hierarchical structure information, which are stored using key-value (KV) storage and a distributed database that supports transactions, respectively.

[0036] Metadata is managed using directory shard groups: Metadata under the same directory is divided into multiple shards, and each shard group is associated with a specific storage tier (high-performance tier or high-capacity tier). Each metadata pool adds a "directory shard group" data structure to record the shard information in that storage pool, and this data structure is part of the directory metadata.

[0037] A sub-item index table is established to record the mapping relationship between directory sub-items and shard groups. The directory shard to which a sub-item belongs is determined through hash calculation. The sub-item index table is managed in memory using the LRU algorithm and stored on disk as multiple data objects, with the corresponding data object obtained through hash calculation.

[0038] Develop a multi-dimensional hierarchical strategy to trigger dynamic migration of metadata between different layers and ensure the rationality of metadata storage.

[0039] Static strategy: Specify the metadata storage layer based on directory path or depth. For example, specify that all metadata under the path / app / hot / is stored in the high-performance layer; set a directory depth threshold, such as storing metadata with a directory depth of more than 5 levels in the large-capacity layer.

[0040] Dynamic strategy: Adjust the storage layer based on the time attributes or access frequency of metadata. For example, metadata created or last accessed within the last 30 days is retained in the high-performance layer, while metadata older than 30 days is migrated to the large-capacity layer; metadata in the large-capacity layer that is accessed more than a threshold (such as 10 times per day) is migrated back to the high-performance layer.

[0041] Scheduled migration: The system periodically scans the high-performance layer metadata and migrates metadata that meets the migration criteria (such as being created more than 90 days ago and not accessed within 30 days) to the large-capacity layer in batches.

[0042] Popularity-triggered migration: For example, if a certain metadata in the high-capacity layer is accessed more than 50 times in 1 hour, it will be triggered to migrate back to the high-performance layer.

[0043] Consistency Guarantee: A distributed locking mechanism is used during the migration process (when MDS is split, existing sharding locks can achieve mutual exclusion between splitting and business operations; this locking mechanism is reused here) to ensure mutual exclusion with business operations and avoid data inconsistency. Write locks are applied to metadata during migration to prevent business modifications; after migration is complete, the locks are released and the index tables are updated.

[0044] The above method enables the rational distribution and dynamic adjustment of metadata across different storage layers. When a business needs to access metadata, the shard group and storage layer where the metadata resides are first determined through the sub-item index table, and then the metadata is retrieved from the corresponding storage layer, improving metadata access efficiency.

[0045] Specifically, such as Figure 5 As shown, an embodiment of this application provides a method for hierarchical storage of metadata, including the following steps:

[0046] Step S1: Construct a three-tier metadata storage architecture consisting of a memory layer, a high-performance storage layer, and a large-capacity storage layer;

[0047] Step S2: In the high-performance storage layer and the large-capacity storage layer, a directory shard group is set up to manage the metadata of the distributed storage. The metadata under the same directory is divided into multiple sub-shards. By establishing a sub-item index table, the sub-shards are associated through the directory-based shard group.

[0048] Step S3: In response to storing sub-shards, obtain the directory path or directory depth of the sub-shard, and store the sub-shard in the high-performance storage layer or the large-capacity storage layer according to the directory path or directory depth of the sub-shard.

[0049] This system employs a three-tiered metadata hierarchical storage architecture, consisting of a memory layer, a high-performance storage layer, and a large-capacity storage layer. Distributed metadata is managed using directory sharding groups in the high-performance and large-capacity storage layers, achieving directory-sharded organization and intelligent tiered storage based on directory paths or depth. By migrating cold metadata to the low-cost large-capacity storage layer, storage costs are significantly reduced; hot metadata is retained in the high-performance layer, ensuring unaffected access performance; directory sharding groups support linear expansion of metadata, meeting large-scale management needs; and differentiated storage improves storage resource utilization and avoids performance waste.

[0050] In this embodiment, a three-tier metadata storage architecture consisting of a memory layer, a high-performance storage layer, and a large-capacity storage layer is constructed, including:

[0051] The memory layer is composed of dynamic random access memory of cluster nodes, and the memory layer is used to cache hot data.

[0052] A high-performance storage layer is composed of all-flash storage media, which is used to store metadata logs.

[0053] A large-capacity storage layer is composed of mixed flash or hard disk storage media, which is used to store cold metadata.

[0054] Specifically, the storage functionality of each layer was optimized by using a combination of different storage media. The memory layer caches hot data, improving data access speed; the high-performance storage layer is suitable for storing rapidly updated metadata logs; and the large-capacity storage layer effectively reduces storage costs and meets the needs of cold data, thereby improving the overall performance of the storage system.

[0055] In this embodiment, the method further includes:

[0056] When new target metadata is available, the target metadata is stored in the memory layer.

[0057] Record the creation time and access frequency of the target metadata stored in the memory layer.

[0058] By caching the target metadata of newly stored data in the memory layer and recording its access information, it is possible to quickly respond to data requests that are frequently accessed, while providing the necessary time and access frequency data support for subsequent data migration, thus optimizing the dynamic management of storage resources.

[0059] In this embodiment, the method further includes:

[0060] The creation time and access frequency of the target metadata are obtained according to a preset cycle;

[0061] If the creation time of the target metadata is greater than or equal to the first threshold, the target metadata will be migrated from the memory layer to the high-performance storage layer.

[0062] If the creation time of the target metadata is greater than or equal to the second threshold, the target metadata will be migrated from the high-performance storage layer to the high-capacity storage layer.

[0063] If the access frequency of the target metadata is less than or equal to the third threshold, the target metadata will be migrated from the memory layer to the mass storage layer or from the mass storage layer to the high-performance storage layer.

[0064] If the access frequency of the target metadata is greater than or equal to the third threshold, the target metadata will be migrated from the mass storage layer to the memory layer or from the high-performance storage layer to the mass storage layer.

[0065] Based on the creation time and access frequency of the target metadata, the system determines whether the migration conditions are met within a preset period and migrates the data from the memory layer to the high-performance storage layer or the large-capacity storage layer.

[0066] The first threshold is preferably 10 days, the second threshold is preferably 30 days, and the third threshold is preferably 10 times per day.

[0067] In this embodiment, the method further includes:

[0068] The popularity value of the target metadata is determined based on its creation time and access frequency.

[0069] If the heat value of the target metadata is less than or equal to the fourth threshold, the target metadata will be migrated from the memory layer to the mass storage layer or from the mass storage layer to the high-performance storage layer.

[0070] If the heat value of the target metadata is greater than or equal to the fourth threshold, the target metadata will be migrated from the memory layer to the mass storage layer or from the mass storage layer to the high-performance storage layer.

[0071] Specifically, the storage location of target metadata is determined by calculating its popularity value, and migration between storage layers is performed based on the popularity value. Dynamically adjusting the storage location of target metadata based on popularity value enables more intelligent data storage management, improving overall system performance and storage resource utilization. High-popularity data is preferentially stored in the memory layer or high-performance storage layer, while low-popularity data is migrated to a large-capacity storage layer, thereby optimizing the adaptability of the storage architecture.

[0072] The fourth threshold is preferably 50 visits within 1 hour.

[0073] In this embodiment, directory shard groups are set up in the high-performance storage layer and the large-capacity storage layer to manage the metadata of distributed storage. The metadata under the same directory is divided into multiple sub-shards, and sub-item index tables are established to associate sub-shards through directory-based shard groups, including:

[0074] Set up at least one metadata pool in the high-performance storage layer and the large-capacity storage layer, and categorize the metadata of each metadata pool in the same directory;

[0075] Set up a directory shard group for each metadata pool, and set the directory shard group to include the main directory and the shard directory;

[0076] When storing metadata, the metadata under the same directory is divided into multiple sub-shards, a sub-item index table is built according to the sub-shards, a shard directory is formed for each sub-shard, the shard information is recorded in the shard directory, and the sub-item index table is associated with the sub-shards;

[0077] Set up the main directory to record shard group information and the global index table, and associate the directory sub-items in the global index table with the shard directories.

[0078] By setting up metadata pools in each storage layer and refining directory shard groups, efficient organization and management of metadata are achieved. Each shard directory contains shard information, and by establishing a global index table associated with the shard directories, metadata storage and retrieval become more efficient, while reducing storage redundancy and access latency.

[0079] In this embodiment, associating directory sub-items in the global index table with sharded directories includes:

[0080] Establish a mapping relationship between directory sub-items in the global index table and sharded directories. The directory sub-item is used to calculate the directory shard group to which it belongs through hash calculation.

[0081] The hash mapping mechanism ensures efficient association between directory sub-items and sharded directories, avoids the complexity of directory management, improves retrieval efficiency and storage organization stability, and ensures fast data access and consistency.

[0082] In this embodiment, when storing metadata, the metadata under the same directory is divided into multiple sub-shards. After establishing a sub-item index table based on the sub-shards, the method further includes:

[0083] The sub-item index table is stored in the sharded directory and managed using the Least Recently Used (LRU) algorithm.

[0084] When storing metadata, multiple data objects are used on the disk, and the corresponding data object is obtained by hash calculation based on the content of the sub-item index table.

[0085] Specifically, the least-nearest-neighbor algorithm optimizes the management of the sub-item index table, making the cache replacement process more efficient. Combined with hash calculation technology, data objects can be efficiently stored and retrieved on disk, improving storage efficiency and reducing data access latency.

[0086] In this embodiment, in response to storing a sub-segment, the directory path or directory depth of the sub-segment is obtained, and the sub-segment is stored in a high-performance storage layer or a large-capacity storage layer according to the directory path or directory depth of the sub-segment, including:

[0087] When storing sub-shards, determine whether the directory path of the sub-shard has been obtained;

[0088] If the directory path of the sub-segment is obtained, all metadata under the directory path is stored in the high-performance storage layer;

[0089] If the directory path of the sub-segment is not obtained, set the directory depth threshold and obtain the directory depth of the sub-segment.

[0090] If the directory depth of a sub-shard is less than the directory depth threshold, the sub-shard will be stored in the high-performance storage layer.

[0091] If the directory depth of a sub-shard is greater than or equal to the directory depth threshold, the sub-shard will be stored in the mass storage layer.

[0092] By determining the storage location based on the directory path or depth of the sub-shard, data storage is made more intelligent and efficient. Sub-shards with shorter path depths are preferentially stored in the high-performance storage layer, thereby improving the response speed of frequently accessed data.

[0093] In this embodiment, the method further includes:

[0094] Determine whether the migration conditions are met based on the access frequency, modification time, and creation time of the metadata stored between the memory layer, high-performance storage layer, and large-capacity storage layer;

[0095] For metadata that meets the migration conditions, the migration operation is triggered by setting an access frequency threshold. During the migration process, a distributed lock mechanism is used for data synchronization to ensure that business operations and access to and modification of metadata do not conflict during the migration.

[0096] Specifically, a distributed lock mechanism ensures data consistency during metadata migration and avoids conflicts between business operations and data migration. This approach enhances the system's support for concurrent operations and guarantees high data availability and consistency.

[0097] The migration criteria are that the creation time is more than 90 days and there has been no access within 30 days.

[0098] In this embodiment, a distributed lock mechanism is used for data synchronization during the migration process, including:

[0099] Apply write locks between the target storage layer and the source storage layer to ensure consistency in accessing and modifying metadata during metadata migration;

[0100] Perform a consistency check before migrating metadata to ensure that the migrated metadata is consistent with the metadata in the source storage layer;

[0101] After the metadata migration is complete, release the lock and update the corresponding sub-item index table and global index table so that subsequent access requests can correctly locate the migrated metadata.

[0102] Specifically, by applying write locks and consistency checks, the integrity and consistency of data are ensured during the migration process. Updating the index table allows the metadata after data migration to be quickly synchronized into the system, guaranteeing the stable operation of the system and the accuracy of the data.

[0103] In this embodiment, the method further includes:

[0104] During metadata storage and migration, data compression is used to reduce storage space, and data is dynamically decompressed during metadata migration to improve storage efficiency and access speed. After metadata migration, data verification is used to check data consistency.

[0105] Among these measures, data compression and decompression technologies were used to reduce storage space usage, while data consistency verification was performed during the migration process to ensure the accuracy and integrity of the migrated data, further optimizing storage efficiency and data consistency.

[0106] In this embodiment, the method further includes:

[0107] A dynamic data migration strategy based on access patterns is established between the memory layer, high-performance storage layer and large-capacity storage layer of the three-level metadata storage architecture. The dynamic data migration strategy obtains multi-dimensional indicators, including access frequency, access interval time, number of modifications, directory popularity distribution and access time window, by statistically analyzing the access behavior of metadata.

[0108] Cluster analysis of multi-dimensional indicators is performed based on machine learning models to predict access popularity trends in the first time period. Based on the access popularity trends, the migration threshold and migration direction of metadata between the memory layer, high-performance storage layer and large-capacity storage layer are adjusted.

[0109] This involves adjusting the migration threshold and direction of metadata between the memory layer, high-performance storage layer, and large-capacity storage layer by adjusting access popularity trends. This enables adaptive optimization of metadata storage layout under different business scenarios, thereby improving the overall access performance and energy efficiency of the system.

[0110] Among them, the machine learning model is a time series-based prediction model, including a Long Short-Term Memory (LSTM) network model or an attention-based prediction model. The model uses access logs, migration records and access paths of metadata as training samples. By learning the time series features, it predicts the future access probability of each directory and file, and dynamically adjusts the heat weight of each directory shard group and the storage layer allocation strategy according to the prediction results. This further reduces the load pressure and energy consumption of the high-performance storage layer while maintaining stable access performance.

[0111] In this embodiment, the method further includes:

[0112] A multi-level index system is maintained in the main directory of the directory shard group. The multi-level index system includes a global index layer, a shard index layer, and an object index layer. The global index layer is used to record the global identification information and logical path mapping relationship of each directory shard group. The shard index layer is used to maintain the correspondence between sub-shards and storage nodes. The object index layer is used to record the physical storage location, verification information, and version number of each metadata object on the disk.

[0113] A multi-level indexing system is used to locate, retrieve, and verify the consistency of metadata across nodes. An atomic transaction mechanism is used when updating the sub-index table and the global index table to ensure the consistency of metadata in high-concurrency scenarios.

[0114] The index update employs an asynchronous commit mechanism based on the transaction log, including:

[0115] When an index changes, the change record is written to the metadata log file in the high-performance storage layer;

[0116] Asynchronous commit operations are triggered periodically in the background to save the changes in the logs to the global index table in batches;

[0117] The asynchronous commit operation employs pre-write verification and multi-replica backup mechanisms to prevent index inconsistencies caused by node anomalies, network interruptions, or power failures, thereby improving the reliability and recovery capabilities of the metadata storage system.

[0118] The aforementioned metadata tiered storage method constructs a three-tiered metadata tiered storage architecture consisting of a memory layer, a high-performance storage layer, and a large-capacity storage layer. Distributed metadata is managed using directory sharding groups in the high-performance and large-capacity storage layers, achieving directory-sharded organization and intelligent tiered storage based on directory paths or depth. By migrating cold metadata to the low-cost large-capacity storage layer, storage costs are significantly reduced; hot metadata is retained in the high-performance layer, ensuring unaffected access performance; directory sharding groups support linear expansion of metadata, meeting large-scale management needs; and differentiated storage improves storage resource utilization and avoids performance waste.

[0119] The present invention has the following beneficial effects:

[0120] Cost optimization: By migrating cold metadata to a low-cost, high-capacity storage layer, the overall storage cost is reduced, which can be more than 40% lower than that of an all-flash storage solution.

[0121] Performance Guarantee: Hot metadata is stored in the high-performance storage layer and memory layer, metadata modification logs are saved in the high-performance layer, and cold metadata can be promoted to the high-performance layer, thus ensuring that metadata access performance is not reduced after tiered storage.

[0122] Enhanced scalability: Adopting a directory sharding and hierarchical storage architecture, it supports linear expansion of metadata volume and can meet the metadata management needs of trillion-level file scenarios.

[0123] Improved resource utilization: Differentiated storage of metadata has been achieved, making full use of high-performance storage resources, avoiding resource waste, and improving the overall utilization of storage resources.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0125] In one embodiment, such as Figure 6 As shown, a metadata hierarchical storage device 10 is provided, including: a storage architecture construction module 1, a directory sharding management module 2, a sharding storage control module 3, a metadata storage and recording module 4, a metadata migration control module 5, a heat value calculation and migration module 6, a migration synchronization and locking mechanism module 7, and a data compression and verification module 8.

[0126] Storage architecture building module 1 is used to build a three-tier metadata storage architecture consisting of a memory layer, a high-performance storage layer, and a large-capacity storage layer.

[0127] The directory sharding management module 2 is used to set up directory sharding groups in the high-performance storage layer and the large-capacity storage layer to manage the metadata of distributed storage. It divides the metadata under the same directory into multiple sub-shards and establishes a sub-item index table to realize the association of sub-shards through directory-based sharding groups.

[0128] The sharded storage control module 3 is used to obtain the directory path or directory depth of the sub-shard in response to storing the sub-shard, and store the sub-shard in the high-performance storage layer or the large-capacity storage layer according to the directory path or directory depth of the sub-shard.

[0129] In this embodiment, a three-tier metadata storage architecture consisting of a memory layer, a high-performance storage layer, and a large-capacity storage layer is constructed, including:

[0130] The memory layer is composed of dynamic random access memory of cluster nodes, and the memory layer is used to cache hot data.

[0131] A high-performance storage layer is composed of all-flash storage media, which is used to store metadata logs.

[0132] A large-capacity storage layer is composed of mixed flash or hard disk storage media, which is used to store cold metadata.

[0133] In this embodiment, the metadata storage and recording module 4 is used for:

[0134] When new target metadata is available, the target metadata is stored in the memory layer.

[0135] Record the creation time and access frequency of the target metadata stored in the memory layer.

[0136] In this embodiment, the metadata migration control module 5 is used for:

[0137] The creation time and access frequency of the target metadata are obtained according to a preset cycle;

[0138] If the creation time of the target metadata is greater than or equal to the first threshold, the target metadata will be migrated from the memory layer to the high-performance storage layer.

[0139] If the creation time of the target metadata is greater than or equal to the second threshold, the target metadata will be migrated from the high-performance storage layer to the high-capacity storage layer.

[0140] If the access frequency of the target metadata is less than or equal to the third threshold, the target metadata will be migrated from the memory layer to the mass storage layer or from the mass storage layer to the high-performance storage layer.

[0141] If the access frequency of the target metadata is greater than or equal to the third threshold, the target metadata will be migrated from the mass storage layer to the memory layer or from the high-performance storage layer to the mass storage layer.

[0142] In this embodiment, the heat value calculation and migration module 6 is used for:

[0143] The popularity value of the target metadata is determined based on its creation time and access frequency.

[0144] If the heat value of the target metadata is less than or equal to the fourth threshold, the target metadata will be migrated from the memory layer to the mass storage layer or from the mass storage layer to the high-performance storage layer.

[0145] If the heat value of the target metadata is greater than or equal to the fourth threshold, the target metadata will be migrated from the memory layer to the mass storage layer or from the mass storage layer to the high-performance storage layer.

[0146] In this embodiment, directory shard groups are set up in the high-performance storage layer and the large-capacity storage layer to manage the metadata of distributed storage. The metadata under the same directory is divided into multiple sub-shards, and sub-item index tables are established to associate sub-shards through directory-based shard groups, including:

[0147] Set up at least one metadata pool in the high-performance storage layer and the large-capacity storage layer, and categorize the metadata of each metadata pool in the same directory;

[0148] Set up a directory shard group for each metadata pool, and set the directory shard group to include the main directory and the shard directory;

[0149] When storing metadata, the metadata under the same directory is divided into multiple sub-shards, a sub-item index table is built according to the sub-shards, a shard directory is formed for each sub-shard, the shard information is recorded in the shard directory, and the sub-item index table is associated with the sub-shards;

[0150] Set up the main directory to record shard group information and the global index table, and associate the directory sub-items in the global index table with the shard directories.

[0151] In this embodiment, associating directory sub-items in the global index table with sharded directories includes:

[0152] Establish a mapping relationship between directory sub-items in the global index table and sharded directories. The directory sub-item is used to calculate the directory shard group to which it belongs through hash calculation.

[0153] In this embodiment, when storing metadata, the metadata under the same directory is divided into multiple sub-shards. After establishing a sub-item index table based on the sub-shards, the method further includes:

[0154] The sub-item index table is stored in the sharded directory and managed using the Least Recently Used (LRU) algorithm.

[0155] When storing metadata, multiple data objects are used on the disk, and the corresponding data object is obtained by hash calculation based on the content of the sub-item index table.

[0156] In this embodiment, in response to storing a sub-segment, the directory path or directory depth of the sub-segment is obtained, and the sub-segment is stored in a high-performance storage layer or a large-capacity storage layer according to the directory path or directory depth of the sub-segment, including:

[0157] When storing sub-shards, determine whether the directory path of the sub-shard has been obtained;

[0158] If the directory path of the sub-segment is obtained, all metadata under the directory path is stored in the high-performance storage layer;

[0159] If the directory path of the sub-segment is not obtained, set the directory depth threshold and obtain the directory depth of the sub-segment.

[0160] If the directory depth of a sub-shard is less than the directory depth threshold, the sub-shard will be stored in the high-performance storage layer.

[0161] If the directory depth of a sub-shard is greater than or equal to the directory depth threshold, the sub-shard will be stored in the mass storage layer.

[0162] In this embodiment, the migration synchronization and locking mechanism module 7 is used for:

[0163] Determine whether the migration conditions are met based on the access frequency, modification time, and creation time of the metadata stored between the memory layer, high-performance storage layer, and large-capacity storage layer;

[0164] For metadata that meets the migration conditions, the migration operation is triggered by setting an access frequency threshold. During the migration process, a distributed lock mechanism is used for data synchronization to ensure that business operations and access to and modification of metadata do not conflict during the migration.

[0165] In this embodiment, a distributed lock mechanism is used for data synchronization during the migration process, including:

[0166] Apply write locks between the target storage layer and the source storage layer to ensure consistency in accessing and modifying metadata during metadata migration;

[0167] Perform a consistency check before migrating metadata to ensure that the migrated metadata is consistent with the metadata in the source storage layer;

[0168] After the metadata migration is complete, release the lock and update the corresponding sub-item index table and global index table so that subsequent access requests can correctly locate the migrated metadata.

[0169] In this embodiment, the data compression and verification module 8 is used for:

[0170] During metadata storage and migration, data compression is used to reduce storage space, and data is dynamically decompressed during metadata migration to improve storage efficiency and access speed. After metadata migration, data verification is used to check data consistency.

[0171] The aforementioned metadata tiered storage device constructs a three-tiered metadata tiered storage architecture consisting of a memory layer, a high-performance storage layer, and a large-capacity storage layer. Distributed metadata is managed using directory sharding groups in the high-performance and large-capacity storage layers, achieving directory-sharded organization and intelligent tiered storage based on directory paths or depth. By migrating cold metadata to the low-cost large-capacity storage layer, storage costs are significantly reduced; hot metadata is retained in the high-performance layer, ensuring unaffected access performance; directory sharding groups support linear expansion of metadata, meeting large-scale management needs; and differentiated storage improves storage resource utilization and avoids performance waste.

[0172] For a description of the features in the corresponding embodiment of the metadata hierarchical storage device, please refer to the relevant description of the corresponding embodiment of the metadata hierarchical storage method, which will not be repeated here.

[0173] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described metadata hierarchical storage method embodiments.

[0174] In one embodiment, the electronic device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores metadata hierarchically stored data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a metadata hierarchical storage method.

[0175] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described metadata hierarchical storage method embodiments at runtime:

[0176] Construct a three-tier metadata storage architecture consisting of a memory layer, a high-performance storage layer, and a large-capacity storage layer;

[0177] In the high-performance storage layer and the large-capacity storage layer, directory shard groups are set up to manage the metadata of distributed storage. The metadata under the same directory is divided into multiple sub-shards, and sub-item index tables are established to realize the association of sub-shards through directory-based shard groups.

[0178] When storing a sub-shard, obtain the directory path or directory depth of the sub-shard, and store the sub-shard in a high-performance storage layer or a large-capacity storage layer according to the directory path or directory depth of the sub-shard.

[0179] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0180] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described metadata hierarchical storage method embodiments:

[0181] Construct a three-tier metadata storage architecture consisting of a memory layer, a high-performance storage layer, and a large-capacity storage layer;

[0182] In the high-performance storage layer and the large-capacity storage layer, directory shard groups are set up to manage the metadata of distributed storage. The metadata under the same directory is divided into multiple sub-shards, and sub-item index tables are established to realize the association of sub-shards through directory-based shard groups.

[0183] When storing a sub-shard, obtain the directory path or directory depth of the sub-shard, and store the sub-shard in a high-performance storage layer or a large-capacity storage layer according to the directory path or directory depth of the sub-shard.

[0184] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described metadata hierarchical storage method embodiments:

[0185] Construct a three-tier metadata storage architecture consisting of a memory layer, a high-performance storage layer, and a large-capacity storage layer;

[0186] In the high-performance storage layer and the large-capacity storage layer, directory shard groups are set up to manage the metadata of distributed storage. The metadata under the same directory is divided into multiple sub-shards, and sub-item index tables are established to realize the association of sub-shards through directory-based shard groups.

[0187] When storing a sub-shard, obtain the directory path or directory depth of the sub-shard, and store the sub-shard in a high-performance storage layer or a large-capacity storage layer according to the directory path or directory depth of the sub-shard.

[0188] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0189] The foregoing has provided a detailed description of a metadata hierarchical storage method, electronic device, storage medium, and program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A metadata tiered storage method, characterized by, The method comprises the following steps: constructing a three-level metadata storage architecture including a memory layer, a high-performance storage layer and a mass storage layer; setting up a directory shard group to manage distributed storage of metadata in the high-performance storage layer and the mass storage layer, dividing metadata under the same directory into multiple sub-shards, and realizing association of the sub-shards by the directory shard group through a sub-item index table; in response to storage of the sub-shards, obtaining a directory path or a directory depth of the sub-shards, and storing the sub-shards in the high-performance storage layer or the mass storage layer according to the directory path or the directory depth of the sub-shards; in response to existence of target metadata newly stored, storing the target metadata in the memory layer; recording a creation time and a frequency of access of the target metadata stored in the memory layer; obtaining the creation time and the frequency of access of the target metadata according to a preset period; judging whether a migration condition is met within the preset period according to the creation time and the frequency of access of the target metadata, and migrating data meeting the migration condition from the memory layer to the high-performance storage layer or the mass storage layer; determining a hotness value of the target metadata according to the creation time and the frequency of access of the target metadata; dynamically adjusting a storage position of the target metadata according to the hotness value; wherein, in response to storage of the sub-shards, obtaining a directory path or a directory depth of the sub-shards, and storing the sub-shards in the high-performance storage layer or the mass storage layer according to the directory path or the directory depth of the sub-shards, comprises: when the sub-shards are stored, judging whether the directory path of the sub-shards is obtained; if the directory path of the sub-shards is obtained, storing all metadata under the directory path in the high-performance storage layer; if the directory path of the sub-shards is not obtained, setting a directory depth threshold, and obtaining a directory depth of the sub-shards; if the directory depth of the sub-shards is less than the directory depth threshold, storing the sub-shards in the high-performance storage layer; if the directory depth of the sub-shards is greater than or equal to the directory depth threshold, storing the sub-shards in the mass storage layer; wherein, the method further comprises: maintaining a multi-level index system in a main directory of the directory shard group, the multi-level index system comprising a global index layer, a shard index layer and an object index layer; wherein the global index layer is used to record global identification information and a logical path mapping relationship of each directory shard group; the shard index layer is used to maintain a corresponding relationship between the sub-shards and storage nodes; and the object index layer is used to record a physical storage position, check information and a version number of each metadata object on a disk; the multi-level index system is used to realize positioning, cross-node retrieval and consistency check of metadata, and an atomic transaction mechanism is used when the sub-item index table and the global index table are updated.

2. The metadata tiering storage method of claim 1, wherein, The three-level metadata storage architecture comprising the memory layer, the high-performance storage layer and the mass storage layer comprises: the memory layer is composed of dynamic random memories of cluster nodes, and the memory layer is used to cache hot metadata; The high-performance storage layer is used for storing metadata logs and is composed of all-flash storage media; The large-capacity storage layer is used for storing cold metadata and is composed of hybrid flash or hard disk storage media.

3. The metadata tiering storage method of claim 1, wherein, The method further includes the following steps: If the creation time of the target metadata is greater than or equal to a first threshold value, the target metadata is migrated from the memory layer to the high-performance storage layer; If the creation time of the target metadata is greater than or equal to a second threshold value, the target metadata is migrated from the high-performance storage layer to the large-capacity storage layer; If the access frequency of the target metadata is less than or equal to a third threshold value, the target metadata is migrated from the memory layer to the large-capacity storage layer or from the high-performance storage layer to the large-capacity storage layer; If the access frequency of the target metadata is greater than the third threshold value, the target metadata is migrated from the large-capacity storage layer to the memory layer or from the large-capacity storage layer to the high-performance storage layer.

4. The metadata tiering storage method of claim 1, wherein, The method further includes the following steps: If the hotness value of the target metadata is less than or equal to a fourth threshold value, the target metadata is migrated from the memory layer to the large-capacity storage layer or from the high-performance storage layer to the large-capacity storage layer; If the hotness value of the target metadata is greater than the fourth threshold value, the target metadata is migrated from the large-capacity storage layer to the memory layer or from the large-capacity storage layer to the high-performance storage layer.

5. The metadata tiering storage method of claim 1, wherein, The method further includes the following steps: The metadata in the high-performance storage layer and the large-capacity storage layer is classified into a plurality of sub-shards, and a sub-item index table is established to associate the sub-shards through the directory shards. At least one metadata pool is set in the high-performance storage layer and the large-capacity storage layer, and the metadata of each metadata pool is classified in the same directory. A directory shard group is set for each metadata pool, and the directory shard group includes a main directory and a shard directory. When the metadata is stored, the metadata in the same directory is classified into a plurality of sub-shards, a sub-item index table is established according to the sub-shards, a shard directory is formed for each sub-shard, the shard information is recorded in the shard directory, and the sub-item index table is associated with the sub-shard.

6. The metadata tiering storage method of claim 5, wherein, The main directory records the shard group information and a global index table, and the directory sub-item in the global index table is associated with the shard directory. The method further includes the following steps: The directory sub-item in the global index table is mapped to the shard directory, and the directory sub-item is obtained by hash calculation.

7. The metadata tiering storage method of claim 5, wherein, The method further comprises: The method further comprises: The method further comprises:

8. The metadata tiering storage method of claim 5, wherein, The method further comprises: The method further comprises: The method further comprises:

9. The metadata tiering storage method of claim 8, wherein, The method further comprises: The method further comprises: The method further comprises: The method further comprises:

10. The metadata tiering storage method of claim 9, wherein, The method further comprises: The method further comprises:

11. An electronic device, comprising: The method further comprises: The method further comprises: The method further comprises:

12. A computer-readable storage medium, characterized in that, The method further comprises:

13. A computer program product comprising a computer program, characterized in that, The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further

Images