A metadata processing method and device based on distributed storage system

Abstract

The embodiment of the present invention provides a metadata processing method and device based on a distributed storage system, which is applied to the field of storage technology, including: caching part of the metadata in the metadata pool of the distributed storage system to a metadata cache layer container, the metadata cache layer container is provided with multiple metadata partitions, when the target metadata partition of the metadata cache layer container hits the target metadata corresponding to the metadata access request, the target metadata is returned to the client, when the target metadata corresponding to the metadata access request is not hit in the multiple metadata partitions of the metadata cache layer container, the target metadata corresponding to the metadata access request is obtained from the metadata pool, the target metadata is cached in the target metadata partition of the metadata cache layer container, and the target metadata is returned to the client, so that a metadata cache layer container is provided in the distributed storage system, and metadata is cached to meet the access requirements for metadata.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a metadata processing method and device based on a distributed storage system. Background Art

[0002] With the development of big data and cloud computing technologies and the popularization of smart devices, the amount and type of data are growing rapidly. Traditional storage systems can no longer meet the needs of large-scale and diversified data storage. Distributed storage systems can provide high-performance, high-reliability and scalable storage solutions.

[0003] In a distributed storage system, metadata management is performed through metadata services to ensure file consistency and reliability. However, with the increase in metadata access requests, existing metadata services are unable to meet the demand. Summary of the invention

[0004] In view of the above problems, a metadata processing method and device based on a distributed storage system is proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A metadata processing method based on a distributed storage system, wherein the distributed storage system is provided with a metadata cache layer container, the method comprising:

[0006] Cache part of the metadata in the metadata pool of the distributed storage system to the metadata cache layer container; wherein the metadata cache layer container is provided with a plurality of metadata partitions;

[0007] When the target metadata partition of the metadata cache layer container hits the target metadata corresponding to the metadata access request, returning the target metadata to the client;

[0008] When the target metadata corresponding to the metadata access request is not hit in the multiple metadata partitions of the metadata cache layer container, the target metadata corresponding to the metadata access request is obtained from the metadata pool, the target metadata is cached in the target metadata partition of the metadata cache layer container, and the target metadata is returned to the client.

[0009] Optionally, it also includes:

[0010] The metadata cached in the metadata cache layer container is dynamically updated.

[0011] Optionally, dynamically updating the metadata cached in the metadata cache layer container includes:

[0012] The weight information of the metadata cached in the metadata cache layer container is determined, and the metadata cached in the metadata cache layer container is dynamically updated according to the weight information.

[0013] Optionally, determining weight information of metadata cached in the metadata cache layer container includes:

[0014] Determining the access frequency and cache hit rate of the metadata cached in the metadata cache layer container;

[0015] According to the access frequency and the cache hit rate, weight information of the metadata cached in the metadata cache layer container is determined.

[0016] Optionally, determining the access frequency and cache hit rate of the metadata cached in the metadata cache layer container includes:

[0017] Obtaining the number of accesses to the metadata cached in the metadata cache layer container and a preset time interval;

[0018] The access frequency of the metadata cached in the metadata cache layer container is determined according to the access count and a preset time interval.

[0019] Optionally, determining the access frequency and cache hit rate of the metadata cached in the metadata cache layer container includes:

[0020] Obtaining the number of accesses and cache hits to the metadata cached in the metadata cache layer container;

[0021] A cache hit rate of the metadata cached in the metadata cache layer container is determined according to the number of accesses and the number of cache hits.

[0022] Optionally, dynamically updating the metadata cached in the metadata cache layer container according to the weight information includes:

[0023] According to the weight information, the metadata cached in the metadata cache layer container is sorted; wherein the weight of the metadata in the first sorting order is greater than the weight of the metadata in the last sorting order;

[0024] A preset number of metadata that are sorted later are removed from the metadata cache layer container.

[0025] Optionally, the distributed storage system is provided with a plurality of storage nodes, and further includes:

[0026] For a metadata write request, a write operation is performed on a first storage node and a second storage node of the distributed storage system according to the metadata write request; wherein the second storage node is a node that backs up the metadata in the first storage node.

[0027] Optionally, for the metadata write request, performing a write operation on the first storage node and the second storage node of the distributed storage system according to the metadata write request includes:

[0028] For a metadata write request, performing a write operation on a first storage node in the distributed storage system according to the metadata write request, and marking the metadata write request as an unconfirmed state;

[0029] Performing a write operation on a second storage node in the distributed storage system according to the metadata write request;

[0030] After all the second storage nodes complete the write operation, the metadata write request is marked as a confirmed state to complete the metadata write request.

[0031] Optionally, before performing a write operation on the second storage node of the distributed storage system according to the metadata write request, the method further includes:

[0032] A negotiation is performed among the plurality of second storage nodes in the distributed storage system to determine an order of write operations among the plurality of second storage nodes.

[0033] Optionally, it also includes:

[0034] A consistent hashing algorithm is used to divide multiple metadata partitions in the metadata cache layer container into multiple data fragments, and store them on different storage nodes in the distributed storage system.

[0035] Optionally, it also includes:

[0036] Receive metadata access requests sent by clients through remote procedure call communication mechanism;

[0037] The returning the target metadata to the client comprises:

[0038] The target metadata is returned to the client through a remote procedure call communication mechanism.

[0039] Optionally, the distributed storage system is provided with a plurality of storage nodes, and further includes:

[0040] Receive data read and write requests sent by the client through the remote procedure call communication mechanism;

[0041] In response to the data read / write request, the data is stored in the form of an object in multiple storage nodes of the distributed storage system, or the data is read from the multiple storage nodes of the distributed storage system.

[0042] Optionally, storing the data in the form of objects in multiple storage nodes of the distributed storage system includes:

[0043] The data to be stored is sliced, and the sliced ​​data is stored in the form of objects in multiple storage nodes of the distributed storage system.

[0044] Optionally, each metadata partition stores metadata in a tree data structure, wherein the tree data structure includes a plurality of tree data structure nodes, and the plurality of tree data structure nodes include a root node and leaf nodes;

[0045] The path from the root node to the leaf node corresponds to a string, and the time complexity of the insertion operation and the search operation in the tree data structure is related to the length of the string.

[0046] Optionally, the distributed storage system provides file system services externally, and the client initiates a metadata access request in the form of accessing a file.

[0047] A metadata processing device based on a distributed storage system, wherein the distributed storage system is provided with a metadata cache layer container, and the device comprises:

[0048] A metadata cache module, used to cache part of the metadata in the metadata pool of the distributed storage system to the metadata cache layer container; wherein the metadata cache layer container is provided with a plurality of metadata partitions;

[0049] A hit processing module, configured to return the target metadata to the client when the target metadata partition of the metadata cache layer container hits the target metadata corresponding to the metadata access request;

[0050] A miss processing module is used to obtain the target metadata corresponding to the metadata access request from the metadata pool when the target metadata corresponding to the metadata access request is not hit in multiple metadata partitions of the metadata cache layer container, cache the target metadata in the target metadata partition of the metadata cache layer container, and return the target metadata to the client.

[0051] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method described above when executed by the processor.

[0052] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0053] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described above.

[0054] The embodiments of the present invention have the following advantages:

[0055] In an embodiment of the present invention, part of the metadata in the metadata pool of the distributed storage system is cached to a metadata cache layer container, and the metadata cache layer container is provided with multiple metadata partitions. When the target metadata partition of the metadata cache layer container hits the target metadata corresponding to the metadata access request, the target metadata is returned to the client. When the target metadata corresponding to the metadata access request is not hit in the multiple metadata partitions of the metadata cache layer container, the target metadata corresponding to the metadata access request is obtained from the metadata pool, the target metadata is cached to the target metadata partition of the metadata cache layer container, and the target metadata is returned to the client. This realizes that a metadata cache layer container is provided in the distributed storage system, and metadata is cached through the metadata cache layer container to meet the access requirements for metadata. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0057] Figure 1 is a schematic diagram of a system architecture provided by some embodiments of the present invention;

[0058] Figure 2 is a schematic diagram of another system architecture provided by some embodiments of the present invention;

[0059] Figure 3 is a flowchart of a metadata processing method based on a distributed storage system provided by some embodiments of the present invention;

[0060] Figure 4 is a schematic diagram of another system architecture provided by some embodiments of the present invention;

[0061] Figure 5 is a structural block diagram of a metadata processing device based on a distributed storage system provided by some embodiments of the present invention;

[0062] Figure 6 is a structural block diagram of an electronic device provided by some embodiments of the present invention;

[0063] Figure 7is a structural block diagram of a storage medium provided by some embodiments of the present invention;

[0064] Figure 8 It is a structural block diagram of a program product provided by some embodiments of the present invention. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] In a distributed storage system, data can be stored on multiple independent storage nodes and managed in a distributed manner, thereby providing a high-performance, high-reliability, and scalable storage solution.

[0067] The design concept of the distributed storage system is based on scalability and fault tolerance. It adopts a distributed architecture, slices data and stores it in the form of objects on multiple nodes, ensures the capacity and load balance of each node, accesses and manages data through the network, and provides a unified storage interface.

[0068] Since the underlying distributed storage system is stored in the form of sliced ​​objects, when providing file system services to the outside world, it is necessary to manage file metadata (such as file name, permissions, and attributes) through metadata services to ensure the consistency and reliability of the file system and provide high-performance file service functions.

[0069] With the large-scale and diversified data storage and the increase in storage costs, metadata operation requests for file attributes, permissions, indexes, etc. account for more than 76% of the massive small file storage IO requests, and the access load and performance of metadata services have been challenged. In massive and large-scale distributed file storage, metadata has become the main bottleneck of distributed storage system IO performance. The main reason is that the horizontal expansion of distributed storage system metadata has limitations, metadata management is too complex, and lacks flexibility and scalability.

[0070] In the related art, in a distributed storage file system, in order to improve the file access efficiency of distributed storage and avoid competition between data access and metadata management, MDS (Metadata Server) is designed to be responsible for the management and storage of metadata. In order to meet the scalability of the distributed storage cluster, multiple MDS service processes are used, and the metadata of the entire file system is partitioned.

[0071] By adopting the manual partitioning method, directory subtrees of different levels are fixedly assigned to each MDS. When the load between MDSs is unbalanced, the subtrees between MDSs are migrated manually according to the load of each node in the MDS cluster, and the subtrees are dynamically adjusted to different MDS nodes to achieve load balancing. MDS nodes with higher loads determine the size of the migrated subtrees based on the popularity of the local directory, and migrate some subtrees to MDS nodes with lighter loads. It can be seen that the load of each MDS node and the location of the subtree are statically fixed. Obviously, this method is more suitable for scenarios where the data location is relatively fixed. However, as the business volume continues to expand and the cluster scale needs to be expanded, it is necessary to manually reallocate the subtrees to the newly added MDS nodes, so the metadata load balancing capability is poor.

[0072] However, the management and maintenance of multiple MDS service processes in a distributed storage cluster is relatively complex. It is necessary to evaluate the number of metadata services based on the distributed underlying infrastructure and manually preset the metadata cache to ensure the performance, consistency, and reliability of the metadata.

[0073] Moreover, metadata multi-service processes are difficult to maintain and require a lot of manpower to adjust and optimize. When there is a high concurrent load of a large number of small files, the metadata cache cannot be elastically expanded in time. Metadata operations need to retrieve metadata from the metadata storage pool through the MDS process, which exacerbates storage IO latency. Although metadata and data management are separated and decoupled for storage, they essentially need to be pre-configured together. Metadata needs to rely on the efficiency and deployment configuration of the backend data storage device. When performance bottlenecks occur in either metadata management or backend storage, storage system performance will drop rapidly.

[0074] In addition, during the dynamic subtree partitioning process, data may need to be migrated between different MDSs, and data migration may lead to performance degradation, increased network overhead, and even the risk of data loss or data inconsistency. Dynamic subtree partitioning increases management complexity, requiring monitoring and adjustment of the load status of different subtrees to ensure the balance and stability of the entire system.

[0075] In the embodiment of the present invention, metadata is managed through the function-as-a-service mode, and the metadata of the distributed storage system is combined with the function-as-a-service mode to provide an efficient, flexible and elastically scalable metadata management solution. Specifically, the function-as-a-service mode is used to process operations on client requests for metadata, trigger metadata update events, and interact with the distributed storage system to ensure the consistency of file system storage IO. Through the embodiment of the present invention, metadata management can be made more flexible and efficient, the processing speed of data operations can be accelerated, the elasticity and scalability of the system can be improved, and the operation and maintenance burden can be reduced.

[0076] Among them, the function-as-a-service model is a cloud computing service model that allows developers to write and deploy functionally independent code, usually running in the form of functions.

[0077] In this model, developers do not need to manage servers, operating systems or network infrastructure. Instead, they focus on writing functions, and the cloud service provider is responsible for managing the underlying infrastructure, scalability and security. Functions written by developers can run as event-driven microservices and are only triggered to execute when needed. This on-demand execution method effectively reduces resource waste while providing a high degree of flexibility and scalability. Functions can be triggered by events to achieve asynchronous processing and fast response. Users can automatically expand the number of instances according to demand to adapt to different load conditions.

[0078] like Figure 1 ,The distributed storage system architecture includes multiple components. ,In the storage pool, the client’s data is stored in slices and ,the default fixed size is 4MB objects. ,The objects are grouped and the group id is the modulo value of the Hash (object ,identifier) ​​and the number of groups, and ,finally stored in the underlying device.

[0079] The distributed storage system architecture includes the following key components:

[0080] 1. Unified, self-controlled, and scalable distributed storage consistency management system: This component is the core component of the distributed storage cluster, which provides distributed storage services in the form of objects. In this component, data is stored as objects, and each object has a unique identifier and related data. It is responsible for storing objects in a distributed manner on each node of the storage cluster, and provides data replication, recovery, load balancing and other functions to ensure data reliability and high-performance access.

[0081] 2. Storage pool: A distributed storage cluster consists of multiple storage nodes, each of which can contain multiple hard disks or storage devices. The storage pool divides storage resources into different pools to meet different storage needs.

[0082] 3. Controllable, scalable, distributed data balanced placement algorithm: This component evenly distributes data objects on each node of the storage cluster to avoid data hotspots and improve system performance.

[0083] 4. Metadata service container module and monitoring service cluster: The metadata service container module is responsible for managing the metadata information of the file system, including the attributes and locations of files and directories. The monitoring service cluster is responsible for monitoring the status and configuration information of the cluster.

[0084] like Figure 2The client of the distributed storage system sends a file system metadata request to the function-as-a-service metadata cache layer. The request communication method adopts the RPC (Remote Procedure Call) mechanism. The function-as-a-service metadata cache layer is executed in the container, and the metadata cache service and other storage services can be run on the distributed storage system at the same time to avoid direct resource competition with each other.

[0085] Among them, the automatic expansion strategy module implements the update mechanism of the cache layer and the parallel processing of metadata, and dynamically expands the metadata in the metadata cache layer container according to the load of metadata requests. In addition, a metadata consistency module is designed to ensure the data consistency and accuracy of user business storage IO and refresh the cached metadata.

[0086] For data read and write IO, according to the path information obtained from metadata, the data is stored in the OSD (Object storage daemon, which is used to store data in the form of objects on the physical disk of each node in the cluster) device of the storage node according to the strategy of the distributed storage system. In addition, through the management of the metadata consistency module, the metadata is persistently stored in the OSD device of the distributed storage node, which uses a high-speed SSD (Solid State Disk) device for storage.

[0087] Specifically, the embodiments of the present invention have the following advantages:

[0088] 1. Performance: Based on the same hardware resources, in a distributed file storage environment, the metadata function as a service management model can automatically scale the number of instances according to the request volume, adapt to different load conditions, achieve elastic expansion, and ensure the thread growth of distributed storage system performance.

[0089] 2. Stability: The present invention is decoupled from the distributed storage system, improves system maintainability and scalability, reduces system complexity, is transparent to business clients, and has stability.

[0090] 3. Security: The components and method modules of the present invention are packaged and can be quickly deployed and started, which reduces the time to go online and the complexity of deployment and is safe.

[0091] 4. Low cost: The metadata hierarchical processing logic and metadata information and structured attribute information storage format proposed in the present invention, and the metadata function as a service management model do not need to consider the distributed storage system infrastructure, but only need to focus on the development of metadata business logic, which can improve the competitiveness of distributed file storage and reduce maintenance costs.

[0092] 5. Compatibility: The present invention can process metadata operations in an event-driven manner, achieve instant response and rapid processing, and has portability and versatility.

[0093] The following further describes the embodiments of the present invention:

[0094] Reference Figure 3 , showing a step flow chart of a metadata processing method based on a distributed storage system provided in some embodiments of the present invention, wherein the distributed storage system is provided with a metadata cache layer container.

[0095] Specifically, the following steps may be included:

[0096] Step 301: cache part of the metadata in the metadata pool of the distributed storage system to the metadata cache layer container; wherein the metadata cache layer container is provided with a plurality of metadata partitions.

[0097] In practical applications, a metadata cache layer container based on function as a service can be set in the distributed storage system, and then part of the metadata in the metadata pool of the distributed storage system can be cached to the meta container.

[0098] In the embodiments of the present invention, full use is made of the function-as-a-service model in cloud computing. First, by utilizing the metadata cache layer container deployment in the function-as-a-service model in cloud computing, the metadata cache directory tree structure can be automatically expanded, quickly and transparently expanded outward to respond to sudden work, so that metadata requests can adapt to changing workloads in real time, and the resource utilization of the distributed storage system is improved, avoiding resource competition with other service modules.

[0099] Moreover, by designing a metadata cache layer container with function as a service, we can solve the problems of complex metadata management mode and multiple dependent conditions. The design does not need to rely on the underlying distributed storage system management server, operating system or network infrastructure, but instead focuses on writing functional functions to design a reasonable metadata management architecture to support seamless expansion of system scale to meet the growing data needs.

[0100] In some embodiments of the present invention, each metadata partition stores metadata in a tree data structure, wherein the tree data structure includes multiple tree data structure nodes, and the multiple tree data structure nodes include a root node and leaf nodes; the path from the root node to the leaf node corresponds to a string, and the time complexity of the insertion operation and the search operation in the tree data structure is related to the length of the string.

[0101] In actual applications, the metadata cache layer container caches metadata information, which is stored in memory through a tree data structure, where each tree data structure node represents a character, the edge represents the connection relationship between characters, and the path from the root node to the leaf node represents a string. The time complexity of the insertion and search operations in this structure is related to the length of the string, usually O(m), where m is the length of the string and O() is a function of the calculation time complexity, which is very fast. Caching metadata through this structure can improve the hit rate of metadata reading operations.

[0102] Step 302: When the target metadata partition of the metadata cache layer container hits the target metadata corresponding to the metadata access request, the target metadata is returned to the client.

[0103] In some embodiments of the present invention, the distributed storage system provides file system services to the outside, and the client initiates a metadata access request in the form of accessing a file.

[0104] In some embodiments of the present invention, it also includes:

[0105] Receive metadata access requests sent by clients through the remote procedure call communication mechanism.

[0106] In some embodiments of the present invention, returning the target metadata to the client includes:

[0107] The target metadata is returned to the client through a remote procedure call communication mechanism.

[0108] In practical applications, the interaction between the client and the distributed storage system can be realized through the RPC communication path to improve the throughput and elasticity of metadata requests. First, design the RPC interface required for metadata request processing, including request type, parameters, and return results. Secondly, write specific metadata request processing logic, including file addition, deletion, modification, and query operations, directory management, metadata retrieval and other functions. Design the corresponding data structure according to each metadata request type, and determine the serialization and deserialization methods of the data to ensure the correctness and integrity of the data during network transmission. The specific process is as follows:

[0109] 1. When the client sends a request, an RPC communication of a TCP connection is established. In the dynamic load, the number of metadata directory tree results of the metadata cache container is instantiated to meet the processing of different communication connection requests.

[0110] 2. The metadata cache container submits the metadata request to the cached directory tree structure metadata. If the requested metadata does not exist in the container, the function program is called to instantiate a new metadata cache.

[0111] Step 303, when the target metadata corresponding to the metadata access request is not hit in the multiple metadata partitions of the metadata cache layer container, the target metadata corresponding to the metadata access request is obtained from the metadata pool, the target metadata is cached in the target metadata partition of the metadata cache layer container, and the target metadata is returned to the client.

[0112] In an embodiment of the present invention, by first determining whether the metadata corresponding to the metadata access request hits the metadata cache layer container, and then determining whether to obtain the target metadata in the metadata cache layer container or in the metadata pool, the problem of low metadata access efficiency is solved. It is necessary to design a suitable metadata access strategy, including metadata caching, metadata distribution, etc., to improve metadata access efficiency.

[0113] like Figure 4 , the processing of metadata access requests is as follows:

[0114] 1. The client initiates metadata request 1 for the file / mnt / tmp / test1.

[0115] 2. Search in the tree structure of the metadata cache layer container, find metadata 2, cache hit, and return the metadata to the client.

[0116] 3. Return metadata directly to the client.

[0117] 4. The client issues another metadata request 2 for the file / mnt / share / test2.

[0118] 5. Cache miss during lookup.

[0119] 6. Get the metadata information of test2 from the metadata pool, create a new directory tree structure cache information of the file in the container, and return it to the client.

[0120] In some embodiments of the present invention, it also includes:

[0121] The metadata cached in the metadata cache layer container is dynamically updated.

[0122] In the embodiment of the present invention, the metadata cached in the metadata cache layer container is dynamically updated to solve the problem of metadata load balancing. In the metadata management process, the metadata load balancing is considered to avoid a single node being overloaded and affecting the performance and stability of the system. It is necessary to monitor and adjust the load distribution of metadata nodes to maintain the balance of the entire system.

[0123] In some embodiments of the present invention, the dynamic updating of the metadata cached in the metadata cache layer container includes: determining weight information of the metadata cached in the metadata cache layer container, and dynamically updating the metadata cached in the metadata cache layer container according to the weight information.

[0124] In some embodiments of the present invention, determining the weight information of the metadata cached in the metadata cache layer container includes: determining the access frequency and cache hit rate of the metadata cached in the metadata cache layer container; and determining the weight information of the metadata cached in the metadata cache layer container based on the access frequency and the cache hit rate.

[0125] In some embodiments of the present invention, determining the access frequency and cache hit rate of the metadata cached in the metadata cache layer container includes: obtaining the number of accesses to the metadata cached in the metadata cache layer container and a preset time interval; and determining the access frequency of the metadata cached in the metadata cache layer container based on the number of accesses and the preset time interval.

[0126] In some embodiments of the present invention, determining the access frequency and cache hit rate of the metadata cached in the metadata cache layer container includes: obtaining the number of accesses and the number of cache hits of the metadata cached in the metadata cache layer container; and determining the cache hit rate of the metadata cached in the metadata cache layer container based on the number of accesses and the number of cache hits.

[0127] In some embodiments of the present invention, dynamically updating the metadata cached in the metadata cache layer container according to the weight information includes:

[0128] According to the weight information, the metadata cached in the metadata cache layer container is sorted; and a preset number of metadata that are sorted later are removed from the metadata cache layer container.

[0129] The weight of the metadata in the first order is greater than the weight of the metadata in the last order.

[0130] In an embodiment of the present invention, an automatic expansion strategy is provided to implement an update mechanism of the cache layer and a parallel processing method of metadata. The metadata cache can improve the throughput of the file system. The number of metadata caches needs to meet the dynamic workload to the maximum extent. Too much metadata cache will occupy memory resources. It is necessary to weigh the metadata cache and memory resource costs. Therefore, an automatic expansion strategy is proposed in an embodiment of the present invention for managing the metadata cache, which is as follows:

[0131] Define metadata access frequency RF(metadata) = number of accesses / time interval;

[0132] Define metadata cache hit rate HR(metadata) = cache hit count / total access count;

[0133] Calculate the metadata weight W(metadata) = RF(metadata) * (1 - HR(metadata)).

[0134] According to the weight W (metadata), several metadata with the lowest weight are selected for elimination, and the cache size is dynamically adjusted as needed to adapt to changes in access patterns.

[0135] 1. Initialize metadata cache: set the initial size of the cache, and initialize metadata access frequency, cache hit rate, weight and other information.

[0136] 2. Monitor metadata access: When metadata is accessed, count the number of accesses and cache hits, and calculate the RF and HR of each metadata.

[0137] 3. Calculate weight: Calculate the weight W (metadata) of metadata according to the formula to decide which metadata will be eliminated. The higher the weight, the more it will be expanded, otherwise, the cache amount will be reduced.

[0138] 4. Automatic expansion strategy: Regularly check the weight of each metadata and expand or reduce the cache size as needed. The metadata with the lowest weight will be replaced or eliminated to make room for metadata with high access frequency.

[0139] 5. Implement cache replacement: Selectively replace metadata based on weight calculation results to maintain efficient use of the cache.

[0140] 6. Dynamic monitoring and adjustment: Continuously monitor metadata access patterns and cache effects, and adjust algorithm parameters and strategies based on actual conditions to optimize cache performance.

[0141] In some embodiments of the present invention, the distributed storage system is provided with a plurality of storage nodes, and further includes:

[0142] For a metadata write request, a write operation is performed on a first storage node and a second storage node of the distributed storage system according to the metadata write request; wherein the second storage node is a node that backs up the metadata in the first storage node.

[0143] In some embodiments of the present invention, for the metadata write request, performing a write operation on the first storage node and the second storage node of the distributed storage system according to the metadata write request includes:

[0144] For a metadata write request, a write operation is performed on the first storage node of the distributed storage system according to the metadata write request, and the metadata write request is marked as an unconfirmed state; a write operation is performed on the second storage node of the distributed storage system according to the metadata write request; after all second storage nodes have completed the write operation, the metadata write request is marked as a confirmed state to complete the metadata write request.

[0145] In some embodiments of the present invention, before performing a write operation on the second storage node of the distributed storage system according to the metadata write request, the method further includes:

[0146] A negotiation is performed among the plurality of second storage nodes in the distributed storage system to determine an order of write operations among the plurality of second storage nodes.

[0147] In actual applications, the following operations can be performed for metadata write operations:

[0148] 1. Send the write request to the candidate primary metadata node, metadata 1, and mark the operation as unconfirmed.

[0149] 2. After receiving the write request, the primary metadata node forwards the write request to all backup replica metadata nodes and requires these nodes to also perform the write operation.

[0150] 3. After receiving the write request, all backup replica metadata nodes negotiate to check the consistency of the order of the write operation and finally reach a consensus.

[0151] 4. Once all backup replica metadata nodes confirm that the write operation is complete, the primary metadata node marks the write request as confirmed, completing the entire write operation.

[0152] In some embodiments of the present invention, it also includes:

[0153] A consistent hashing algorithm is used to divide multiple metadata partitions in the metadata cache layer container into multiple data fragments, and store them on different storage nodes in the distributed storage system.

[0154] In actual applications, the parent directory ID is used as input, and the corresponding hash value (partition) is calculated through the consistent hashing algorithm. Each partition corresponds to the directory tree structure metadata 1 to n, and each partition is used to determine the storage location of the metadata. When nodes are dynamically increased or decreased, the characteristics of consistent hashing can ensure that most of the data is still mapped to the original nodes, reducing the overhead caused by data migration and redistribution. Consistent hashing can simplify metadata management and location search, reduce the amount of information that needs to be tracked, and improve the scalability and flexibility of the system. Then, each deployed metadata cache container instance is responsible for the partition of the metadata namespace. Finally, the client operates the metadata according to the partition.

[0155] In practical applications, a consistent hashing algorithm is used to divide metadata partitions 1 to n in the metadata cache layer container into multiple fragments and store them on different nodes in the distributed storage system. The hash function is hash = H (metadata partitions 1 to n), and the metadata partitions 1 to n in the metadata cache layer container are mapped to a position on a hash ring = hash mod n. The value range of the hash ring is generally a fixed interval, and n is the number of nodes in the distributed storage cluster of the present invention.

[0156] In some embodiments of the present invention, the distributed storage system is provided with a plurality of storage nodes, and further includes:

[0157] Through the remote procedure call communication mechanism, a data read and write request sent by a client is received; in response to the data read and write request, data is stored in the form of an object in multiple storage nodes of the distributed storage system, or data is read from multiple storage nodes of the distributed storage system.

[0158] In some embodiments of the present invention, storing data in the form of objects in multiple storage nodes of the distributed storage system includes: slicing the data to be stored, and storing the sliced ​​data in the form of objects in multiple storage nodes of the distributed storage system.

[0159] In a distributed storage system, a distributed architecture is adopted to slice data and store them in the form of objects on multiple nodes, ensuring the capacity and load balance of each node, accessing and managing data through the network, and providing a unified storage interface.

[0160] Since the underlying distributed storage system is stored in the form of sliced ​​objects, when providing file system services to the outside world, it is necessary to manage file metadata (such as file name, permissions, and attributes) through metadata services to ensure the consistency and reliability of the file system and provide high-performance file service functions.

[0161] In an embodiment of the present invention, part of the metadata in the metadata pool of the distributed storage system is cached to a metadata cache layer container, and the metadata cache layer container is provided with multiple metadata partitions. When the target metadata partition of the metadata cache layer container hits the target metadata corresponding to the metadata access request, the target metadata is returned to the client. When the target metadata corresponding to the metadata access request is not hit in the multiple metadata partitions of the metadata cache layer container, the target metadata corresponding to the metadata access request is obtained from the metadata pool, the target metadata is cached to the target metadata partition of the metadata cache layer container, and the target metadata is returned to the client. This realizes that a metadata cache layer container is provided in the distributed storage system, and metadata is cached through the metadata cache layer container to meet the access requirements for metadata.

[0162] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0163] Reference Figure 5 , shows a schematic diagram of the structure of a metadata processing device based on a distributed storage system provided by some embodiments of the present invention, wherein the distributed storage system is provided with a metadata cache layer container, which may specifically include the following modules:

[0164] The metadata cache module 501 is used to cache part of the metadata in the metadata pool of the distributed storage system to the metadata cache layer container; wherein the metadata cache layer container is provided with a plurality of metadata partitions;

[0165] A hit processing module 502 is used to return the target metadata to the client when the target metadata partition of the metadata cache layer container hits the target metadata corresponding to the metadata access request;

[0166] The miss processing module 503 is used to obtain the target metadata corresponding to the metadata access request from the metadata pool when the target metadata corresponding to the metadata access request is not hit in multiple metadata partitions of the metadata cache layer container, cache the target metadata in the target metadata partition of the metadata cache layer container, and return the target metadata to the client.

[0167] Optionally, it also includes:

[0168] The metadata cached in the metadata cache layer container is dynamically updated.

[0169] Optionally, dynamically updating the metadata cached in the metadata cache layer container includes:

[0170] The weight information of the metadata cached in the metadata cache layer container is determined, and the metadata cached in the metadata cache layer container is dynamically updated according to the weight information.

[0171] Optionally, determining weight information of metadata cached in the metadata cache layer container includes:

[0172] Determining the access frequency and cache hit rate of the metadata cached in the metadata cache layer container;

[0173] According to the access frequency and the cache hit rate, weight information of the metadata cached in the metadata cache layer container is determined.

[0174] Optionally, determining the access frequency and cache hit rate of the metadata cached in the metadata cache layer container includes:

[0175] Obtaining the number of accesses to the metadata cached in the metadata cache layer container and a preset time interval;

[0176] The access frequency of the metadata cached in the metadata cache layer container is determined according to the access count and a preset time interval.

[0177] Optionally, determining the access frequency and cache hit rate of the metadata cached in the metadata cache layer container includes:

[0178] Obtaining the number of accesses and cache hits to the metadata cached in the metadata cache layer container;

[0179] A cache hit rate of the metadata cached in the metadata cache layer container is determined according to the number of accesses and the number of cache hits.

[0180] Optionally, dynamically updating the metadata cached in the metadata cache layer container according to the weight information includes:

[0181] According to the weight information, the metadata cached in the metadata cache layer container is sorted; wherein the weight of the metadata in the first sorting order is greater than the weight of the metadata in the last sorting order;

[0182] A preset number of metadata that are sorted later are removed from the metadata cache layer container.

[0183] Optionally, the distributed storage system is provided with a plurality of storage nodes, and further includes:

[0184] For a metadata write request, a write operation is performed on a first storage node and a second storage node of the distributed storage system according to the metadata write request; wherein the second storage node is a node that backs up the metadata in the first storage node.

[0185] Optionally, for the metadata write request, performing a write operation on the first storage node and the second storage node of the distributed storage system according to the metadata write request includes:

[0186] For a metadata write request, performing a write operation on a first storage node in the distributed storage system according to the metadata write request, and marking the metadata write request as an unconfirmed state;

[0187] Performing a write operation on a second storage node in the distributed storage system according to the metadata write request;

[0188] After all the second storage nodes complete the write operation, the metadata write request is marked as a confirmed state to complete the metadata write request.

[0189] Optionally, before performing a write operation on the second storage node of the distributed storage system according to the metadata write request, the method further includes:

[0190] A negotiation is performed among the plurality of second storage nodes in the distributed storage system to determine an order of write operations among the plurality of second storage nodes.

[0191] Optionally, it also includes:

[0192] A consistent hashing algorithm is used to divide multiple metadata partitions in the metadata cache layer container into multiple data fragments, and store them on different storage nodes in the distributed storage system.

[0193] Optionally, it also includes:

[0194] Receive metadata access requests sent by clients through remote procedure call communication mechanism;

[0195] The returning the target metadata to the client comprises:

[0196] The target metadata is returned to the client through a remote procedure call communication mechanism.

[0197] Optionally, the distributed storage system is provided with a plurality of storage nodes, and further includes:

[0198] Receive data read and write requests sent by the client through the remote procedure call communication mechanism;

[0199] In response to the data read and write request, the data is stored in the form of an object in multiple storage nodes of the distributed storage system, or the data is read from the multiple storage nodes of the distributed storage system.

[0200] Optionally, storing the data in the form of objects in multiple storage nodes of the distributed storage system includes:

[0201] The data to be stored is sliced, and the sliced ​​data is stored in the form of objects in multiple storage nodes of the distributed storage system.

[0202] Optionally, each metadata partition stores metadata in a tree data structure, wherein the tree data structure includes a plurality of tree data structure nodes, and the plurality of tree data structure nodes include a root node and leaf nodes;

[0203] The path from the root node to the leaf node corresponds to a string, and the time complexity of the insertion operation and the search operation in the tree data structure is related to the length of the string.

[0204] Optionally, the distributed storage system provides file system services externally, and the client initiates a metadata access request in the form of accessing a file.

[0205] In an embodiment of the present invention, part of the metadata in the metadata pool of the distributed storage system is cached to a metadata cache layer container, and the metadata cache layer container is provided with multiple metadata partitions. When the target metadata partition of the metadata cache layer container hits the target metadata corresponding to the metadata access request, the target metadata is returned to the client. When the target metadata corresponding to the metadata access request is not hit in the multiple metadata partitions of the metadata cache layer container, the target metadata corresponding to the metadata access request is obtained from the metadata pool, the target metadata is cached to the target metadata partition of the metadata cache layer container, and the target metadata is returned to the client. This realizes that a metadata cache layer container is provided in the distributed storage system, and metadata is cached through the metadata cache layer container to meet the access requirements for metadata.

[0206] Reference Figure 6 , shows an electronic device provided by some embodiments of the present invention, including a processor 601, a memory 602, and a computer program stored in the memory 602 and capable of running on the processor 601, and the above method is implemented when the computer program is executed by the processor.

[0207] Reference Figure 7 , shows a computer-readable storage medium 700 provided in some embodiments of the present invention, on which a computer program is stored, and the computer program implements the above method when executed by a processor.

[0208] Reference Figure 8 , shows a computer program product 800 provided by some embodiments of the present invention, including a computer program, which implements the above method when executed by a processor.

[0209] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0210] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0211] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0212] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0213] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0214] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0215] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0216] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0217] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0218] The metadata processing method and device based on a distributed storage system are introduced in detail above. The principle and implementation mode of the present invention are explained by using specific examples in this article. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A metadata processing method based on a distributed storage system, characterized in that: The distributed storage system is provided with a metadata cache layer container, and the method comprises: Cache part of the metadata in the metadata pool of the distributed storage system to the metadata cache layer container; wherein the metadata cache layer container is provided with a plurality of metadata partitions; When the target metadata partition of the metadata cache layer container hits the target metadata corresponding to the metadata access request, returning the target metadata to the client; When the target metadata corresponding to the metadata access request is not hit in the multiple metadata partitions of the metadata cache layer container, the target metadata corresponding to the metadata access request is obtained from the metadata pool, the target metadata is cached in the target metadata partition of the metadata cache layer container, and the target metadata is returned to the client; The distributed storage system is provided with a plurality of storage nodes, and further includes: For a metadata write request, a write operation is performed on a first storage node and a second storage node of the distributed storage system according to the metadata write request; wherein the second storage node is a node that backs up the metadata in the first storage node; Wherein, for the metadata write request, performing a write operation on the first storage node and the second storage node of the distributed storage system according to the metadata write request includes: For a metadata write request, performing a write operation on a first storage node in the distributed storage system according to the metadata write request, and marking the metadata write request as an unconfirmed state; Performing a write operation on a second storage node in the distributed storage system according to the metadata write request; After all the second storage nodes complete the write operation, the metadata write request is marked as a confirmed state to complete the metadata write request.

2. The method according to claim 1, characterized in that Also includes: The metadata cached in the metadata cache layer container is dynamically updated.

3. The method according to claim 2, characterized in that The dynamically updating the metadata cached in the metadata cache layer container includes: The weight information of the metadata cached in the metadata cache layer container is determined, and the metadata cached in the metadata cache layer container is dynamically updated according to the weight information.

4. The method according to claim 3, characterized in that The determining weight information of the metadata cached in the metadata cache layer container includes: Determining the access frequency and cache hit rate of the metadata cached in the metadata cache layer container; According to the access frequency and the cache hit rate, weight information of the metadata cached in the metadata cache layer container is determined.

5. The method according to claim 4, characterized in that The determining the access frequency and cache hit rate of the metadata cached in the metadata cache layer container includes: Obtaining the number of accesses to the metadata cached in the metadata cache layer container and a preset time interval; The access frequency of the metadata cached in the metadata cache layer container is determined according to the access count and a preset time interval.

6. The method according to claim 4, characterized in that The determining the access frequency and cache hit rate of the metadata cached in the metadata cache layer container includes: Obtaining the number of accesses and cache hits to the metadata cached in the metadata cache layer container; A cache hit rate of the metadata cached in the metadata cache layer container is determined according to the number of accesses and the number of cache hits.

7. The method according to claim 3, characterized in that The step of dynamically updating the metadata cached in the metadata cache layer container according to the weight information includes: According to the weight information, the metadata cached in the metadata cache layer container is sorted; wherein the weight of the metadata in the first sorting order is greater than the weight of the metadata in the last sorting order; A preset number of metadata that are sorted later are removed from the metadata cache layer container.

8. The method according to claim 1, characterized in that: Before performing a write operation on the second storage node of the distributed storage system according to the metadata write request, the method further includes: A negotiation is performed among the plurality of second storage nodes in the distributed storage system to determine an order of write operations among the plurality of second storage nodes.

9. The method according to claim 1, characterized in that: Also includes: A consistent hashing algorithm is used to divide multiple metadata partitions in the metadata cache layer container into multiple data fragments, and store them on different storage nodes in the distributed storage system.

10. The method according to any one of claims 1 to 7, characterized in that: Also includes: Receive metadata access requests sent by clients through remote procedure call communication mechanism; The returning the target metadata to the client comprises: The target metadata is returned to the client through a remote procedure call communication mechanism.

11. The method according to any one of claims 1 to 7, characterized in that: The distributed storage system is provided with a plurality of storage nodes, and further comprises: Receive data read and write requests sent by the client through the remote procedure call communication mechanism; In response to the data read and write request, the data is stored in the form of an object in multiple storage nodes of the distributed storage system, or the data is read from the multiple storage nodes of the distributed storage system.

12. The method according to claim 11, characterized in that The storing of data in the form of objects in the multiple storage nodes of the distributed storage system includes: The data to be stored is sliced, and the sliced ​​data is stored in the form of objects in multiple storage nodes of the distributed storage system.

13. The method according to claim 1, characterized in that Each metadata partition stores metadata in a tree data structure, wherein the tree data structure includes a plurality of tree data structure nodes, and the plurality of tree data structure nodes include a root node and leaf nodes; The path from the root node to the leaf node corresponds to a string, and the time complexity of the insertion operation and the search operation in the tree data structure is related to the length of the string.

14. The method according to claim 1, characterized in that The distributed storage system provides file system services externally, and the client initiates a metadata access request in the form of accessing files.

15. A metadata processing device based on a distributed storage system, characterized in that: The distributed storage system is provided with a metadata cache layer container, and the device comprises: A metadata cache module, used to cache part of the metadata in the metadata pool of the distributed storage system to the metadata cache layer container; wherein the metadata cache layer container is provided with a plurality of metadata partitions; A hit processing module, configured to return the target metadata to the client when the target metadata partition of the metadata cache layer container hits the target metadata corresponding to the metadata access request; a miss processing module, configured to, when the target metadata corresponding to the metadata access request is not hit in the multiple metadata partitions of the metadata cache layer container, obtain the target metadata corresponding to the metadata access request from the metadata pool, cache the target metadata in the target metadata partition of the metadata cache layer container, and return the target metadata to the client; The distributed storage system is provided with a plurality of storage nodes, and further includes: For a metadata write request, a write operation is performed on a first storage node and a second storage node of the distributed storage system according to the metadata write request; wherein the second storage node is a node that backs up the metadata in the first storage node; Wherein, for the metadata write request, performing a write operation on the first storage node and the second storage node of the distributed storage system according to the metadata write request includes: For a metadata write request, performing a write operation on a first storage node in the distributed storage system according to the metadata write request, and marking the metadata write request as an unconfirmed state; Performing a write operation on a second storage node in the distributed storage system according to the metadata write request; After all the second storage nodes complete the write operation, the metadata write request is marked as a confirmed state to complete the metadata write request.

16. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 14 when executed by the processor.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

18. A computer program product, characterized in that It comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images