Metadata management method and device, electronic equipment and storage medium

Abstract

The invention discloses a metadata management method and device, electronic equipment and a storage medium, and relates to the technical field of data storage, and the method comprises the steps: receiving a metadata operation request sent by a client, determining a target logic partition based on a mapping relation between an identifier and the logic partition, and determining a target physical sub-book group through a mapping relationship between the logic partition and the physical sub-book group, and selecting a target storage node to process the metadata operation request based on the storage node state information in the target physical sub-book group. According to the method, the most suitable storage node can be intelligently determined to execute the metadata operation by introducing a mapping mechanism between the identifier and the logic partition as well as between the logic partition and the physical sub-book group and combining the state information of the storage node; the problems that in traditional metadata management, a single server becomes a performance bottleneck and data searching and maintaining efficiency is low are solved, and the technical effects of improving system performance, improving data processing efficiency and enhancing system reliability are achieved.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a metadata management method, device, electronic device, and storage medium. Background Art

[0002] With the rapid development of information technology, data volumes have skyrocketed, and distributed storage systems have become the mainstream solution for handling large-scale data. In distributed storage systems, metadata management is crucial because it records key information such as data location, size, and permissions, directly impacting system performance and reliability.

[0003] However, traditional metadata management approaches typically rely on a single metadata server to handle all requests. This can easily lead to performance bottlenecks when dealing with large amounts of data, impacting system responsiveness. Furthermore, the lack of effective data partitioning and replica management mechanisms makes data search and maintenance inefficient, and system reliability difficult to guarantee. Summary of the Invention

[0004] The present application provides a metadata management method, device, electronic device and storage medium to at least solve the performance bottleneck, search and maintenance efficiency problems existing in traditional metadata management methods.

[0005] The present application provides a metadata management method, including: receiving a metadata operation request sent by a client, the metadata operation request including a target identifier of target metadata; determining a target logical partition corresponding to the target identifier based on a first mapping relationship between the identifier and the logical partition; determining a target physical replica group corresponding to the target logical partition based on a second mapping relationship between the logical partition and the physical replica group, the physical replica group being a set of replica storage nodes corresponding to the logical partition; extracting a target storage node from the target physical replica group based on status information of multiple storage nodes in the target physical replica group, and sending the metadata operation request to the target storage node, so that the target storage node executes the metadata operation request.

[0006] The present application also provides a metadata management device, including: a receiving module, used to receive a metadata operation request sent by a client, the metadata operation request including a target identifier of the target metadata; a first determination module, used to determine the target logical partition corresponding to the target identifier based on a first mapping relationship between the identifier and the logical partition; a second determination module, used to determine the target physical replica group corresponding to the target logical partition based on a second mapping relationship between the logical partition and the physical replica group, the physical replica group being a set of replica storage nodes corresponding to the logical partition; a management module, used to extract the target storage node from the target physical replica group based on status information of multiple storage nodes in the target physical replica group, and send the metadata operation request to the target storage node, so that the target storage node executes the metadata operation request.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any one of the above-mentioned metadata management methods when executing the computer program.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above-mentioned metadata management methods are implemented.

[0009] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned metadata management methods when executed by a processor.

[0010] Through this application, due to the use of a mapping relationship between identifiers and logical partitions, logical partitions and physical replica groups, and the dynamic selection of target storage nodes to process metadata operation requests in combination with the status information of storage nodes, it is possible to solve technical problems such as performance bottlenecks, low data search efficiency and uneven storage node load in traditional metadata management, thereby achieving the technical effects of improving system performance, improving data processing efficiency, enhancing system reliability and load balancing. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A flowchart of a metadata management method provided in an embodiment of the present application;

[0013] Figure 2A schematic diagram of each storage node carrying resource unit information provided in an embodiment of the present application;

[0014] Figure 3 A schematic diagram of the storage node store3 provided in an embodiment of the present application;

[0015] Figure 4 A flowchart of another metadata management method provided in an embodiment of the present application;

[0016] Figure 5 A schematic diagram of a Region list provided in an embodiment of the present application;

[0017] Figure 6 A schematic diagram of a physical replica relationship chart provided in an embodiment of the present application;

[0018] Figure 7 A structural block diagram of the metadata management device provided in an embodiment of the present application;

[0019] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0021] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0022] The rapid development of information technology and the explosive growth of massive amounts of data have driven the widespread adoption of distributed storage systems. Metadata, as the core information that records data storage location, attributes, and permissions, directly determines system performance and reliability. Traditional metadata management solutions face significant bottlenecks when dealing with large-scale data. A single metadata server can easily become a performance bottleneck due to surging data volumes, leading to delayed system responses. Furthermore, the lack of intelligent data partitioning and management mechanisms makes data retrieval and maintenance inefficient, making it difficult to adapt to modern storage requirements.

[0023] Among related technologies, Region technology improves management efficiency and system scalability by dividing metadata into multiple regions, but its data replica management capabilities are weak. While Ceph PG (Placement Group) replica management technology optimizes data distribution and replica reliability through the CRUSH (Controlled Replication Under Scalable Hashing) algorithm, it suffers from rigid metadata organization and an inability to support range queries. Both technologies face common challenges: different storage protocols require independent maintenance of shard metadata, resulting in wasted resources; the Region model lacks dynamic load balancing capabilities; and Ceph's fixed hash space design necessitates global data migration for capacity expansion. Furthermore, metadata fragmentation caused by Region splits further constrains system performance optimization, necessitating a more flexible metadata architecture design.

[0024] In view of this, the technical solution of the present invention proposes a distributed storage metadata management method that combines Region technology with Ceph PG replica management, aiming to improve the efficiency, reliability and scalability of metadata management, thereby overcoming the technical difficulties faced by related distributed storage systems in dynamic data sharding management, such as metadata expansion, low interval search efficiency, and difficulty in ensuring the consistency of split / merge operations.

[0025] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0026] According to an embodiment of the present invention, an embodiment of a metadata management method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] In this embodiment, a metadata management method is provided, which can be used in electronic devices, such as servers. Figure 1 is a flowchart of a metadata management method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0028] Step S101: receiving a metadata operation request sent by a client, where the metadata operation request includes a target identifier of target metadata.

[0029] A metadata operation request refers to a request initiated by the client to perform a specific operation on metadata, including actions such as storing and querying metadata. For example, the client requests to store the metadata of a certain file. The target metadata refers to the metadata object involved in the current operation request, such as the metadata information of a certain file that needs to be stored or queried. The target identifier refers to the unique identifier of the target metadata. Specifically, the client initiates metadata operations (such as storage, query, etc.) by sending a request to the Region Manager in the electronic device. The metadata operation request must carry the target identifier of the target metadata, such as the unique key value of a file. This process is implemented through network communication between the client and the Region Manager.

[0030] Step S102: determining a target logical partition corresponding to the target identifier based on a first mapping relationship between the identifier and the logical partition.

[0031] An identifier is a unique identifier for metadata. A logical partition (Region) is a metadata logical partition used to manage metadata within a specific range. The first mapping relationship is the mapping relationship between an identifier and a logical partition. The target logical partition is the logical partition determined by the first mapping relationship based on the target identifier, such as Region 0. Specifically, upon receiving a target identifier from a client, the Region Manager uses a matching algorithm to determine the target logical partition to which the target identifier belongs.

[0032] Step S103: Based on the second mapping relationship between the logical partition and the physical replica group, determine the target physical replica group corresponding to the target logical partition, where the physical replica group is a set of replica storage nodes corresponding to the logical partition.

[0033] A physical replica group (SG) is a physical unit of data distribution, responsible for managing a group of storage nodes and their replicas. Each physical replica group contains multiple storage nodes (e.g., store0, store3). The second mapping relationship refers to the mapping relationship between logical partitions and physical replica groups. The target physical replica group refers to the physical replica group of the target logical partition determined by the second mapping relationship, such as SG0. Specifically, each logical partition corresponds to a physical replica group, and the target physical replica group corresponding to the target logical partition is determined based on the corresponding relationship between the logical partition and the physical replica group.

[0034] Step S104 , based on the status information of multiple storage nodes in the target physical replica group, extract the target storage node from the target physical replica group, and send the metadata operation request to the target storage node, so that the target storage node executes the metadata operation request.

[0035] Multiple storage nodes refer to a set of specific storage nodes in the target physical replica group that are responsible for storing the metadata replica of the logical partition. For example, SG0 may include store3, store0, and store4. The target storage node refers to a specific storage node selected from the target physical replica group based on status information (such as activity, load balancing), which is used to perform the current metadata operation. For example, store3 in SG0 is selected as the master replica node. Specifically, the storage node status information in the target physical replica group may include, for example, activity, load conditions, replica roles (such as master replica or slave replica), etc. Based on these states, the master replica node (Leader) or a node with lower load is selected as the target storage node. For example, SG0 = {store3, store0, store4}, extract the master replica (such as store3) to perform the operation.

[0036] Among them, Figure 2 and Figure 3 As shown, each storage node carries resource unit information, including IP address information, port information (such as port0), disk information, status information, etc.

[0037] The metadata management method provided by an embodiment of the present invention achieves efficient positioning and dynamic management of metadata through a two-layer mapping mechanism of logical partitions and physical replica groups. The target logical partition is quickly determined based on the identifier, reducing global search overhead. Through state awareness of the physical replica group, the optimal storage node is dynamically selected to perform operations, which not only improves load balancing capabilities but also enhances system fault tolerance. At the same time, the multi-node redundant design of the replica group ensures high data availability, and the hierarchical structure supports flexible expansion and maintenance. Therefore, while ensuring metadata operation efficiency, this method significantly improves the reliability and scalability of the system.

[0038] In this embodiment, a metadata management method is provided, which can be used in electronic devices, such as servers. Figure 4 is a flowchart of a metadata management method according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0039] Step S201: Receive a metadata operation request sent by a client. The metadata operation request includes a target identifier of the target metadata. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0040] Step S202: determining a target logical partition corresponding to the target identifier based on a first mapping relationship between the identifier and the logical partition.

[0041] Specifically, the target identifier of the target metadata is the target key value corresponding to the target metadata. The above step S202 includes:

[0042] Step S2021: Obtain the key value interval corresponding to each logical partition.

[0043] The target key value (Key) refers to the unique key value corresponding to the target metadata. Each logical partition is responsible for managing a specific key value range (i.e., key value interval), which is defined by the start key value (start key) and the end key value (end key). For example, Figure 5 As shown, Region 0 manages the key range [a, b), and Region 1 manages [b, c]. Specifically, key ranges are dynamically created based on configuration during system initialization and stored in the Region Manager's Region List. To obtain a key range, a client or Region Manager can query this list to obtain the key ranges for all logical partitions.

[0044] Step S2022: Match the target key value with the start key value and the end key value corresponding to each key value interval to determine the target key value interval in which the target key value falls.

[0045] The starting key value refers to the lower limit value of the key value interval managed by the logical partition (such as a). The ending key value refers to the upper limit value of the key value interval managed by the logical partition (such as b). The target key value interval refers to the specific key value range that the target key value falls into. For example, if the target key value is "a0", then its corresponding target key value interval is [a, b), and the corresponding logical partition is Region0. Specifically, the Region manager traverses the key value intervals of all logical partitions, and compares the target key value with the starting key value and ending key value of each interval in turn. If the target key value satisfies the condition of starting key value ≤ target key value ≤ ending key value, then the interval is the target key value interval. This process can be quickly located through binary search or lexicographic order to improve matching efficiency.

[0046] Step S2023: determine the logical partition corresponding to the target key value interval as the target logical partition.

[0047] Once it is determined that the target key value falls into a certain key value range, the Region Manager directly locates the logical partition corresponding to the range (such as Region0) through the pre-stored mapping relationship (such as the Region list) and identifies it as the target logical partition of the current operation.

[0048] The metadata management method provided by the embodiment of the present invention realizes the rapid positioning of logical partitions through a precise matching mechanism of key value intervals. Based on the predefined starting key value and ending key value range, the target key value is directly mapped to the corresponding logical partition, avoiding global traversal or complex calculations, and significantly improving query efficiency. At the same time, the interval division is clear, supporting efficient range query operations, and ensuring the uniqueness of key values ​​within the partition, reducing the risk of data conflicts. This structured partition management method simplifies the organization and maintenance of metadata and provides a clear boundary basis for dynamically expanding partitions.

[0049] Step S203: Based on the second mapping relationship between the logical partition and the physical replica group, determine the target physical replica group corresponding to the target logical partition, where the physical replica group is a set of replica storage nodes corresponding to the logical partition.

[0050] Specifically, the above step S203 includes:

[0051] Step S2031: Determine a target physical replica group identifier corresponding to the target logical partition based on a third mapping relationship between the logical partition and the physical replica group identifier.

[0052] The physical replica group identifier (SG ID) is a symbol used to uniquely identify a physical replica group, such as "SG0" or "SG1". Each logical partition is mapped to a specific SGID to determine the storage location of its data replica. The third mapping relationship is the correspondence between the logical partition and the physical replica group identifier. For example, Region0 is mapped to SG0, and Region1 is mapped to SG1. The target physical replica group identifier is a physical replica group identifier associated with the target logical partition determined by the third mapping relationship. Specifically, after determining the target logical partition (such as Region0), the Region manager directly obtains the physical replica group identifier (such as SG0) bound to the Region by querying a pre-stored mapping table or database (such as the "Region-SGID" correspondence table). This identifier is the target physical replica group identifier.

[0053] Step S2032: Search the preset physical replica relationship graph for the target physical replica group corresponding to the target physical replica group identifier.

[0054] The second mapping relationship includes the third mapping relationship.

[0055] The physical replica relationship graph is a global table or database that records the specific storage node information of each physical replica group. For example, Figure 6As shown, SG0 corresponds to the storage node list {store3, store0, store4}. Specifically, by entering the target physical replica group identifier (such as SG0), a query operation is directly performed in the physical replica relationship graph to match the corresponding storage node set, which is the target physical replica group.

[0056] The metadata management method provided by the embodiment of the present invention realizes the flexible management and rapid positioning of physical resources through a hierarchical mapping mechanism between logical partitions and physical replica group identifiers. Based on the direct mapping relationship between logical partitions and replica group identifiers, the replica group matching process is simplified and the intermediate computing overhead is reduced. The replica group information is centrally maintained through a preset physical replica relationship graph, which supports dynamic adjustment of storage node configuration without the need to reconstruct the mapping logic, significantly improving the maintainability of the system. The hierarchical design (identifier mapping and graph query) decouples the logical and physical layers, enhances scalability, and reduces coupling risks, providing an efficient and unified control basis for dynamic expansion or fault migration of replica groups.

[0057] Step S204 , based on the status information of multiple storage nodes in the target physical replica group, extract the target storage node from the target physical replica group, and send the metadata operation request to the target storage node, so that the target storage node executes the metadata operation request.

[0058] Specifically, the above step S204 includes:

[0059] Step S2041: using the status information of multiple storage nodes in the target physical replica group, determine the storage node with the best node status among the multiple storage nodes; determine the storage node with the best node status among the multiple storage nodes as the target storage node.

[0060] Storage node status information can include metrics such as load, network latency, storage space, and response time. Specifically, this data is collected through real-time monitoring or a heartbeat mechanism, and a comprehensive evaluation is performed based on a pre-set weighting strategy (e.g., prioritizing the node with the lowest load or fastest response). For example, if the CPU utilization of store0 is 20%, store1 is 50%, and store2 is 10%, then store2 is determined to be the node with the best status. Ultimately, the optimal node (e.g., store2) is determined as the target storage node.

[0061] Step S2042: Send the metadata operation request to the target storage node, so that the target storage node accesses the storage directory corresponding to the numbering information based on the numbering information of the target logical partition and executes the metadata operation request under the storage directory.

[0062] The number information refers to the unique identifier of the logical partition (such as "Region0" or "Region1"), which is used to locate the storage directory of the partition in the storage node. The storage directory is the file path on the storage node used to store metadata for a specific logical partition, such as Figure 3 As shown, it is located in the disk information (disk 1). For example, if the target logical partition is numbered Region0, the storage node stores its metadata in the / data / Region0 directory. Specifically, after determining the target storage node, the client or Region manager sends a metadata operation request (such as storage or query) to the node through a network protocol (such as HTTP or RPC). After receiving the request, the target storage node locates the corresponding storage directory (such as / data / Region0) in the local file system based on the numbering information of the target logical partition (such as Region0), and performs the requested operation (such as writing or reading metadata files) in the directory. After the operation is completed, the node returns the result to the client, completing the entire process.

[0063] For example, to store metadata, the client sends a metadata storage request to the Region Manager, including the metadata key and data content. The Region Manager searches for the corresponding region based on the key. If it finds the region, it retrieves the storage group (SG) replica information related to the metadata within that region. The client then sends the metadata to the node specified in the SG replica information for storage. After receiving the metadata, the storage node performs data persistence operations and returns the storage result to the client.

[0064] During metadata lookup, the client sends a metadata lookup request to the Region Manager, along with the metadata key. The Region Manager searches for the corresponding region based on the key. If a region is found, it retrieves the storage group (SG) replica information related to the metadata within that region. The client retrieves the metadata from the node specified in the SG replica information. After receiving the lookup request, the storage node queries its locally stored metadata and returns the result to the client.

[0065] Furthermore, the detection module in the electronic device monitors changes in SG replica information in real time, such as node failures and the addition of new nodes. The detection module sends an update request to the Region Manager, carrying the changed SG number and new replica information. The Region Manager updates the SG replica information for the corresponding Region based on the request. Subsequent metadata storage and lookup operations use the updated SG replica information.

[0066] The metadata management method provided by the embodiments of the present invention achieves optimal resource scheduling by dynamically evaluating the real-time status information of storage nodes. Based on comprehensive indicators such as node health and load level, it automatically selects the storage nodes with the best status, ensuring that metadata operations are preferentially processed by high-performance nodes, significantly improving request response speed and throughput. Furthermore, a state-aware mechanism avoids faulty or heavily loaded nodes, enhancing system fault tolerance and stability. This intelligent node selection strategy not only optimizes resource utilization but also reduces operation latency, providing efficient and adaptive metadata service assurance for distributed storage environments. A precise mapping mechanism between logical partition numbers and storage directories enables rapid and targeted execution of metadata operations. Directly locating the corresponding physical storage directory based on the logical partition number avoids the overhead of path resolution or dynamic addressing, significantly shortening operation response time. The clear directory structure design simplifies metadata storage and access logic, reducing operational complexity and error risks. Furthermore, the static binding of numbers and directories supports on-demand expansion of partition capacity. Adding new partitions simply requires creating directories according to the numbering rules, without adjusting existing logic, enhancing system flexibility and maintainability. This direct mapping mechanism provides the underlying support for efficient and stable metadata management.

[0067] In some optional embodiments, the above-mentioned metadata management method also includes: monitoring the operating status of each replica storage node corresponding to the physical replica group; for any target replica storage node among each replica storage node, if the operating status of the target replica storage node indicates a node failure, the target replica storage node in the physical replica group is reallocated, and the second mapping relationship is updated using the reallocated target replica storage node.

[0068] The operating status of each replica storage node refers to the real-time health indicators of the replica storage node, which may include CPU usage, memory usage, remaining disk space, network latency, response time, etc. The target replica storage node refers to the specific node that needs to be replaced during the exception handling process. Specifically, the health indicators of the storage nodes (such as CPU usage, memory usage, disk space, network latency) are collected regularly in real time, and the online status of the nodes is detected through a heartbeat mechanism or active polling. If the node does not respond continuously or the indicator exceeds the threshold (such as CPU>90%), it is judged to be abnormal. When a node abnormality is detected, a new node (such as store5) is selected according to the preset replica strategy (such as load balancing) to migrate the data copy. Replace the faulty node with a new node in the physical replica relationship diagram, and synchronously update the second mapping relationship (such as Region0→SG0={store0, store4, store5}).

[0069] The metadata management method provided by the embodiment of the present invention achieves efficient fault tolerance and self-healing capabilities of storage nodes through real-time monitoring and dynamic reallocation mechanisms. Continuous monitoring of the operating status of replica nodes can quickly locate faulty nodes and avoid data unavailability due to single point failures. Automatically triggering node reallocation and updating mapping relationships ensures that the replica group always maintains effective redundancy, ensuring data reliability and service continuity. The dynamic synchronization mechanism of the mapping relationship enables the system to adapt to node changes without manual intervention, significantly improving operation and maintenance efficiency and system robustness. This closed-loop fault-tolerant design provides stable and automated fault recovery capabilities for distributed storage environments.

[0070] In some optional embodiments, the above-mentioned metadata management method also includes: when a network interruption is detected between the client and the target storage node, triggering the client request retry mechanism; if the number of retry transmissions corresponding to the client request retry mechanism exceeds a preset threshold, generating a network abnormality alarm information; and / or, for any target replica storage node among each replica storage node, when it is detected that the replica data of the target replica storage node is lost, determining a new replica storage node based on the preset replica strategy, and using the new replica storage node to synchronously update the second mapping relationship.

[0071] The client request retry mechanism is an automatic recovery strategy when network communication fails. The preset threshold is a predefined critical value used to trigger specific operations. The network anomaly alarm information is a prompt message generated when a persistent network failure is detected. Specifically, the client automatically retries when the request times out or a network error occurs. For example, it retries after the first failure at an interval of 1 second, and retries up to 5 times (preset threshold). During the retry period, it continuously attempts to reestablish the connection with the target node. When the number of retries exceeds the threshold (such as 5 times), an alarm message is generated, including the IP address of the failed node, timestamp, and error type (such as "store3 network interruption"), and the administrator is reminded through the log or notification platform.

[0072] The default replication policy is the rule for selecting a new node when a replica is lost or a node fails. Examples include load balancing (preferring the node with the lowest CPU / memory usage) and geographic proximity (selecting the node closest to the client). Specifically, if monitoring and verifying the replica status reveals an incomplete replica, store 6 is selected from a healthy node according to the default replication policy. Data is copied from other replicas to store 6, and the physical replica group is updated to {store 0, store 3, store 6}, which is then synchronized to the Region Manager.

[0073] The metadata management method provided by the embodiment of the present invention comprehensively enhances the robustness and data integrity of the system through network retries and automatic repair of replicas. Request retries are automatically triggered when the network is interrupted, reducing the impact of transient failures on services. The alarm threshold mechanism is combined to avoid resource waste caused by invalid retries, and actively notify operation and maintenance intervention. When a replica is lost, a new node is dynamically allocated based on the strategy and the mapping relationship is updated synchronously to ensure that data redundancy always meets the preset requirements and avoid the risk of data unavailability due to missing replicas. It not only guarantees the ultimate reachability of client requests, but also maintains the high reliability of metadata storage, significantly improving the system's adaptability and continuous service capabilities for complex abnormal scenarios.

[0074] In some optional implementations, the metadata management method further includes dynamic partition reorganization and intelligent load prediction mechanisms.

[0075] A partition load-aware engine and adaptive reorganization module are deployed in electronic devices. By monitoring the metadata operation frequency, data volume growth trends, and storage node performance indicators (including I / O throughput, CPU load matrix, and network latency heat map) of each logical partition in real time, a partition health scoring system based on a time series prediction model is constructed. When the access QPS of a specific logical partition continuously exceeds a dynamic threshold or a local hotspot appears on a storage node, an intelligent partition splitting protocol is triggered. The original key-value interval is split into multiple sub-intervals based on load distribution characteristics (for example, the key-value interval [a,b] of the original Region0 is split into the interval [a,k) where hotspot data is concentrated, generating new intervals [a,k) and [k,b)). Simultaneously, the optimal node combination is automatically calculated based on the topology-aware strategy of the physical replica group, and a new physical replica group is created with distributed transaction consistency guarantees. A two-phase migration technique is used during the splitting process: non-hotspot data is asynchronously replicated in the background, followed by millisecond-level switching through incremental log balancing. During this period, client requests are seamlessly routed to the new logical partition via the redirected metadata service. After the reorganization is completed, the first mapping relationship and the second mapping relationship are synchronously updated through atomic transactions, and a versioned mapping snapshot is generated to achieve failure rollback; in addition, the system has a pre-embedded machine learning-driven capacity planning interface, which actively generates partition pre-splitting suggestions by analyzing historical load cycle patterns, and completes elastic resource expansion before business peaks.

[0076] The metadata management method provided by the embodiment of the present invention realizes the intelligent elastic scaling and high-availability service guarantee of ultra-large-scale metadata clusters through a partition load-aware engine and an adaptive reorganization mechanism. The identification of load hotspots based on real-time monitoring triggers millisecond-level partition splitting, and the dynamic construction of replica groups based on precise key-value interval splitting and topology awareness is combined with two-stage migration technology to achieve service-insensitive expansion. At the same time, relying on the machine learning-driven capacity planning interface to predict load trends, proactively pre-split partitions before business peaks, and form a closed-loop optimization with the node fault recovery process, ultimately achieving load-sensitive adaptive expansion, fault-insensitive service continuity, and maximized resource utilization, providing high-throughput, low-latency underlying support for massive metadata management.

[0077] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0078] The embodiment of the present application also provides a metadata management device, such as Figure 7 Shown, including:

[0079] Receiving module 301, configured to receive a metadata operation request sent by a client, the metadata operation request including a target identifier of target metadata;

[0080] A first determining module 302 is configured to determine a target logical partition corresponding to a target identifier based on a first mapping relationship between the identifier and the logical partition;

[0081] A second determining module 303 is configured to determine a target physical replica group corresponding to the target logical partition based on a second mapping relationship between the logical partition and the physical replica group, where the physical replica group is a set of replica storage nodes corresponding to the logical partition;

[0082] The management module 304 is configured to extract a target storage node from the target physical replica group based on status information of multiple storage nodes in the target physical replica group, and send a metadata operation request to the target storage node so that the target storage node executes the metadata operation request.

[0083] In some optional implementations, the first determining module 302 includes:

[0084] The acquisition submodule is used to obtain the key value interval corresponding to each logical partition;

[0085] A matching submodule is used to match the target key value with the start key value and the end key value corresponding to each key value interval, and determine the target key value interval into which the target key value falls;

[0086] The first determining submodule is configured to determine the logical partition corresponding to the target key value interval as the target logical partition.

[0087] In some optional implementations, the second determining module 303 includes:

[0088] a second determining submodule, configured to determine a target physical replica group identifier corresponding to the target logical partition based on a third mapping relationship between the logical partition and the physical replica group identifier; wherein the second mapping relationship includes the third mapping relationship;

[0089] The search submodule is used to search for a target physical replica group corresponding to the target physical replica group identifier from a preset physical replica relationship graph.

[0090] In some optional implementations, the management module 304 includes:

[0091] A third determining submodule is configured to determine a storage node with the best node status among the multiple storage nodes by using status information of the multiple storage nodes in the target physical replica group;

[0092] The fourth determining submodule is configured to determine a storage node with the best node status among the multiple storage nodes as a target storage node.

[0093] In some optional implementations, the management module 304 further includes:

[0094] The management submodule is used to send the metadata operation request to the target storage node, so that the target storage node accesses the storage directory corresponding to the numbering information based on the numbering information of the target logical partition and executes the metadata operation request under the storage directory.

[0095] In some optional implementations, the metadata management device further includes:

[0096] The monitoring submodule is used to monitor the operating status of each replica storage node corresponding to the physical replica group;

[0097] The first allocation submodule is used to reallocate the target replica storage node in the physical replica group for any target replica storage node among the replica storage nodes, if the operating status of the target replica storage node indicates a node failure, and update the second mapping relationship using the reallocated target replica storage node.

[0098] In some optional implementations, the metadata management device further includes:

[0099] The trigger submodule is used to trigger the client request retry mechanism when a network interruption is detected between the client and the target storage node;

[0100] A generating submodule, configured to generate a network abnormality alarm message if the number of retry transmissions corresponding to the client request retry mechanism exceeds a preset threshold; and / or,

[0101] The second allocation submodule is used to determine a new replica storage node based on a preset replica strategy for any target replica storage node among the replica storage nodes when it is detected that the replica data of the target replica storage node is lost, and to synchronously update the second mapping relationship using the new replica storage node.

[0102] For the description of the features in the embodiment corresponding to the metadata management device, please refer to the relevant description of the embodiment corresponding to the metadata management method, and no further details will be given here.

[0103] The embodiment of the present application also provides an electronic device, such as Figure 8 As shown, it includes a memory 10 and a processor 20. The memory 10 stores a computer program, and the processor 20 is configured to run the computer program to execute the steps in any of the above metadata management method embodiments.

[0104] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned metadata management method embodiments when running.

[0105] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0106] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned metadata management method embodiments are implemented.

[0107] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned metadata management method embodiments are implemented.

[0108] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0109] The above describes in detail the metadata management method, device, electronic device, and storage medium provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core concept of this application. It should be noted that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A metadata management method, characterized in that: include: receiving a metadata operation request sent by a client, wherein the metadata operation request includes a target identifier of target metadata; Determine a target logical partition corresponding to the target identifier based on a first mapping relationship between the identifier and the logical partition; Determine, based on a second mapping relationship between the logical partition and the physical replica group, a target physical replica group corresponding to the target logical partition, the physical replica group being a set of replica storage nodes corresponding to the logical partition; Based on status information of multiple storage nodes in the target physical replica group, a target storage node is extracted from the target physical replica group, and the metadata operation request is sent to the target storage node, so that the target storage node executes the metadata operation request.

2. The metadata management method according to claim 1, characterized in that: The target identifier of the target metadata is the target key value corresponding to the target metadata; The determining, based on the first mapping relationship between the identifier and the logical partition, a target logical partition corresponding to the target identifier includes: Obtaining the key value interval corresponding to each of the logical partitions; Matching the target key value with the start key value and the end key value corresponding to each key value interval to determine the target key value interval into which the target key value falls; The logical partition corresponding to the target key value interval is determined as the target logical partition.

3. The metadata management method according to claim 1 or 2, characterized in that: The determining, based on the second mapping relationship between the logical partition and the physical replica group, a target physical replica group corresponding to the target logical partition includes: Determining a target physical replica group identifier corresponding to the target logical partition based on a third mapping relationship between the logical partition and the physical replica group identifier; Searching for a target physical replica group corresponding to the target physical replica group identifier from a preset physical replica relationship graph; The second mapping relationship includes the third mapping relationship.

4. The metadata management method according to claim 1, wherein: The extracting a target storage node from the target physical replica group based on status information of a plurality of storage nodes in the target physical replica group includes: Determining a storage node with the best node status among the plurality of storage nodes by using status information of the plurality of storage nodes in the target physical replica group; A storage node with the best node status among the multiple storage nodes is determined as the target storage node.

5. The metadata management method according to claim 1 or 4, characterized in that: Sending the metadata operation request to the target storage node so that the target storage node executes the metadata operation request includes: The metadata operation request is sent to the target storage node, so that the target storage node accesses the storage directory corresponding to the numbering information based on the numbering information of the target logical partition and executes the metadata operation request under the storage directory.

6. The metadata management method according to claim 1, wherein: Also includes: Monitoring the running status of each replica storage node corresponding to the physical replica group; For any target replica storage node among the replica storage nodes, if the operating status of the target replica storage node indicates a node failure, the target replica storage node in the physical replica group is reallocated, and the second mapping relationship is updated using the reallocated target replica storage node.

7. The metadata management method according to claim 6, characterized in that: Also includes: When a network interruption is detected between the client and the target storage node, a client request retry mechanism is triggered; If the number of retry transmissions corresponding to the client request retry mechanism exceeds a preset threshold, a network abnormality alarm message is generated; and / or, For any target replica storage node among the replica storage nodes, when it is detected that the replica data of the target replica storage node is lost, a new replica storage node is determined based on the preset replica strategy, and the second mapping relationship is synchronously updated using the new replica storage node.

8. A metadata management device, characterized in that: include: A receiving module, configured to receive a metadata operation request sent by a client, wherein the metadata operation request includes a target identifier of target metadata; A first determining module, configured to determine a target logical partition corresponding to the target identifier based on a first mapping relationship between the identifier and the logical partition; A second determining module is configured to determine a target physical replica group corresponding to the target logical partition based on a second mapping relationship between the logical partition and the physical replica group, where the physical replica group is a set of replica storage nodes corresponding to the logical partition; A management module is used to extract a target storage node from the target physical copy group based on status information of multiple storage nodes in the target physical copy group, and send the metadata operation request to the target storage node so that the target storage node executes the metadata operation request.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the metadata management method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the metadata management method according to any one of claims 1 to 7 are implemented.

Images