A metadata request concurrency control method and system for distributed block storage

Abstract

The application discloses a kind of metadata request concurrent control method and system for distributed block storage, first according to the attribute information and service logic information of distributed block storage system metadata, metadata is grouped, and the root node between the sub-level tree structure based on grouping is obtained Service logic information establishes to obtain metadata structure tree, then based on metadata structure tree, tree-shaped lock mechanism is constructed, and fine-grained lock management is realized;For the time point and RPC request type of each user RPC request, transaction ID is created for request, by analyzing the user RPC request corresponding to each transaction ID, the specific node that request needs to process is determined, then from the total control node of tree-shaped lock, all relevant root node paths are traversed downwards, and locking operation is performed step by step, to ensure the availability of metadata service, the fine-grained characteristics of tree-shaped lock allow concurrent execution of requests on different sub-trees, improve the concurrency and throughput of metadata service.

Description

Technical Field

[0001] This invention relates to the field of distributed block storage, and in particular to a method and system for concurrent control of metadata requests for distributed block storage. Background Technology

[0002] A distributed block storage system refers to a data storage system that distributes data across multiple independent servers to achieve high reliability, scalability, and high performance. Distributed block storage systems provide virtual block devices for user access. Users can mount these virtual block devices to virtual machines or compute nodes using storage protocols (such as iSCSI, NVMe-over-fabrics, etc.). Typically, the data carried by these virtual block devices is divided into several fixed-size data blocks for management. The metadata service of the distributed block storage system needs to manage the metadata of these access block devices and data block metadata, usually through internal service calls.

[0003] A typical metadata management service can be divided into, for example: Figure 10 In the management module, multiple modules will call each other when processing metadata requests. For example, when a request to create an iSCSILUN is received, the access block device management module needs to create the corresponding block device, create the corresponding number of data blocks and associate them with the block device, and finally allocate these data blocks to the disks of the corresponding servers.

[0004] As the workload of distributed block storage systems increases, disk expansion or node expansion inevitably leads to an increase in the amount of system metadata. The metadata service will also manage more data and be requested more frequently. To ensure the high performance of the storage system, the metadata service needs to be able to:

[0005] High-priority requests can be processed first. Request blocking on the I / O path will significantly impact system performance.

[0006] To improve throughput, non-mutually exclusive metadata requests should be processed in parallel as much as possible.

[0007] Current concurrency control strategies for metadata management services in distributed block storage systems at the module level have the following problems:

[0008] 1. It cannot distinguish priorities and can only rely on the order of network requests for processing. Low-priority services may occupy corresponding request processing resources, resulting in increased queuing delays for high-latency sensitive requests on the IO path and affecting the overall system performance.

[0009] 2. The request locks the module processor when making module calls. When a large number of requests come in at the same time, it is difficult to achieve concurrency, which makes it impossible for the metadata management service to make efficient use of computing resources.

[0010] 3. The timing of locking during request processing is related to the timing of module calls, which makes metadata services prone to deadlocks. When implementing new services, developers face a heavy mental burden in order to avoid deadlocks. Summary of the Invention

[0011] The purpose of this invention is to provide a method and system for concurrent control of metadata requests for distributed block storage, which solves the aforementioned technical problems pointed out in the prior art.

[0012] This invention provides a method for concurrent control of metadata requests in distributed block storage, comprising the following steps:

[0013] Based on the attribute information and business logic information corresponding to each metadata in the distributed block storage system, the metadata in the distributed block storage system is grouped to establish multiple sub-level tree structures; a common central control node is established for all sub-level tree structures based on the business logic information between the root nodes of each sub-level tree structure to obtain the metadata tree structure.

[0014] A tree lock is obtained by constructing a tree structure based on the metadata structure tree;

[0015] Each node in the tree lock includes a read-write lock, a set of transaction IDs currently queued for read, and a set of transaction IDs currently queued for write;

[0016] Obtain the user's RPC request, its time point, and its type; create a transaction ID for the user's RPC request based on its time point and type.

[0017] Parse the user RPC requests corresponding to each transaction ID to obtain the processing object node for each user RPC request; traverse all root nodes corresponding to the processing object node, starting from the master node of the tree lock and proceeding downwards level by level to perform locking processing by combining the transaction ID of the user RPC request and deadlock avoidance processing operations.

[0018] Preferably, the process of traversing all root nodes corresponding to the processing object node, starting from the master node of the tree lock and proceeding downwards level by level, and combining the transaction ID of the user's RPC request with deadlock avoidance operations to perform locking, further includes the following steps:

[0019] Based on the data memory snapshot corresponding to the transaction ID of the current user's RPC request, the processing object node repeatedly runs the current user's RPC request for a preset time period of waiting time. After the current user's RPC request is completed, the corresponding processing object node is unlocked.

[0020] Preferably, the step of repeatedly running the current user's RPC request based on the data memory snapshot corresponding to the transaction ID of the current user's RPC request by the processing object node after a preset time period of waiting time, and unlocking the corresponding processing object node after completing the current user's RPC request, includes the following operation steps:

[0021] Get the transaction ID of the current user's RPC request;

[0022] Obtain a memory snapshot of the current RPC request data based on the transaction ID; then apply a lock to the tree lock based on the memory snapshot.

[0023] The locking process includes read lock acquisition and write lock acquisition;

[0024] Determine whether the locking was successful; if so, verify whether the locking process meets the expected result based on the data memory snapshot; if so, process the current user's RPC request, and after processing the current user's RPC request, perform unlocking processing on the processing object node.

[0025] If locking fails, the locking process will be re-executed after a preset time period until the node to be locked is unlocked.

[0026] Preferably, the data memory snapshot includes metadata attribute information and request semantic information.

[0027] Preferably, the read lock acquisition process includes the following steps:

[0028] Traverse all nodes in the tree lock and determine whether the current node is the root node;

[0029] If the current node is determined to be the root node, then it is further determined whether there are any user RPC requests other than the current write request in the processing queue of the current node; if it is determined that there are no user RPC requests other than the current write request in the processing queue of the current node, then the current transaction ID is added to the set of transaction IDs in the queue of read requests, and the read blocking handling of the read-write lock is called, and the lock acquisition success result is output.

[0030] If it is determined that there are user RPC requests other than the current write request in the current node's processing queue, then it is further determined whether the transaction ID of the current user RPC request is less than the transaction ID of user RPC requests other than the current write request in the write request processing queue; if so, the current transaction ID is added to the set of transaction IDs in the read request queue, and the read blocking handling of the read-write lock is called, and the lock acquisition success result is output.

[0031] If it is determined that the transaction ID of the current user's RPC request is not less than the transaction ID of the user's RPC requests in the write request processing queue other than the current write request, then determine whether a read lock has been acquired on the parent node of the current root node.

[0032] If it is determined that a read lock has been acquired on the parent node of the current root node, then the read lock on the parent node of the current root node is released, and a result indicating that the lock acquisition failed is returned; if it is determined that a read lock has not been acquired on the parent node of the current root node, then a result indicating that the lock acquisition failed is returned directly.

[0033] If the current node is not the root node, then acquire a read lock on the parent node of the current node; check whether the read lock acquisition on the parent node of the current node was successful.

[0034] If it is determined that the parent node of the current node has successfully acquired the read lock, then return to the above processing queue for judgment until the output shows either a lock acquisition failure result or a lock acquisition success result.

[0035] Preferably, the write lock acquisition process includes the following steps:

[0036] Traverse all nodes in the tree lock and determine whether the current node is the root node;

[0037] If the current node is determined to be the root node, then it is further determined whether there are any user RPC requests other than the current read / write request in the processing queue of the current node; if it is determined that there are no user RPC requests other than the current read / write request in the processing queue of the current node, then the current transaction ID is added to the set of transaction IDs in the write request queue, and the write blocking handling of the read / write lock is called, and the lock acquisition success result is output.

[0038] If it is determined that there are user RPC requests other than the current write request in the current node's processing queue, then it is further determined whether the transaction ID of the current user RPC request is less than the transaction ID of user RPC requests other than the current write request in the write request processing queue, and at the same time, it is also determined whether the transaction ID of the current user RPC request is less than the transaction ID of user RPC requests other than the current write request in the read request processing queue.

[0039] If the conditions are met simultaneously that the transaction ID of the current user's RPC request is less than the transaction ID of user RPC requests in the write request processing queue (excluding the current write request) and the transaction ID of the current user's RPC request is less than the transaction ID of user RPC requests in the read request processing queue (excluding the current write request), then it is determined whether a read lock has been acquired on the parent node of the current root node. If so, the current transaction ID is added to the set of transaction IDs in the write request queue, and the write blocking handling of the read-write lock is invoked, outputting the result of successful lock acquisition.

[0040] If it is determined that a read lock has been acquired on the parent node of the current root node, then the read lock on the parent node of the current root node is released, and a result indicating that the lock acquisition failed is returned; if it is determined that a read lock has not been acquired on the parent node of the current root node, then a result indicating that the lock acquisition failed is returned directly.

[0041] If the current node is not the root node, then acquire a read lock on the parent node of the current node; check whether the read lock acquisition on the parent node of the current node was successful.

[0042] If it is determined that the parent node of the current node has successfully acquired the read lock, then return to the above processing queue for judgment until the output shows either a lock acquisition failure result or a lock acquisition success result.

[0043] Preferably, the unlocking process for the target node includes the following steps:

[0044] Determine if the currently processed object node is the root node;

[0045] If the node being processed is determined to be the root node, the unlocking instruction for the read / write lock is invoked, and the current transaction ID is removed from the set of transaction IDs queued for read / write requests.

[0046] If it is determined that the currently processed object node is not the root node, then obtain the parent node of the currently processed object node, call the unlocking instruction of the read / write lock, and remove the current transaction ID from the set of transaction IDs queued for read / write requests.

[0047] Preferably, after traversing all root nodes corresponding to the processing object node, and sequentially performing locking processing from the master node of the tree lock downwards, combining the transaction ID of the user's RPC request and deadlock avoidance processing operations, the process further includes...

[0048] If the duration of the lock on a node in the structure tree by the transaction ID reaches a duration threshold within a preset time period, then the node in the structure tree will be unlocked.

[0049] Accordingly, the present invention also proposes a concurrent control system for metadata requests for distributed block storage, including a structure tree establishment module, a tree lock establishment module, a transaction ID creation module, and a locking processing module;

[0050] The structure tree building module is used to group the metadata in the distributed block storage system according to the attribute information and business logic information corresponding to each metadata in the distributed block storage system to obtain multiple sub-structure trees; and to build a common master control node for all sub-tree structures according to the business logic information between the root nodes of each sub-tree structure to obtain the metadata structure tree.

[0051] The tree lock creation module is used to create a tree lock based on the metadata structure tree;

[0052] Each node in the tree lock includes a read-write lock, a set of transaction IDs currently queued for read, and a set of transaction IDs currently queued for write;

[0053] The transaction ID creation module is used to obtain the user's RPC request, the time point of the user's RPC request, and the RPC request type; and to create a transaction ID for the user's RPC request based on the time point and RPC request type.

[0054] The locking processing module is used to parse the user RPC requests corresponding to each transaction ID to obtain the processing object node of each user RPC request; based on all root nodes corresponding to the processing object node, it traverses and sequentially performs locking processing by combining the transaction ID of the user RPC request and deadlock avoidance processing operations, starting from the master control node of the tree lock.

[0055] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0056] Analysis of the metadata request concurrency control method and system for distributed block storage provided by this invention reveals that, in practical applications, all metadata of the distributed block storage system is first acquired, including attribute information (such as type, location, etc.) and business logic information (such as association relationships, access modes, etc.) for each metadata. Then, the metadata is grouped based on this information, with each group forming a sub-tree structure (e.g., Common tree, iSCSI tree, NVM tree, etc.) reflecting the internal logical hierarchy of the metadata. Finally, based on the business logic information (such as protocol associations, topology relationships) between the root nodes of each sub-tree structure... This approach establishes a common central control node for all subtrees, integrating them into a complete metadata structure tree. This transforms metadata management from coarse-grained (module-level) to fine-grained (object-level) management. The tree structure visually represents the business logic relationships of metadata, and the central control node provides a global view, enabling subsequent concurrency control to be based on the entire tree structure rather than processing each module in isolation. This lays the foundation for building a tree-based lock. Furthermore, the tree structure allows for parallel processing of requests from different subtrees, improving the concurrency performance of metadata services; for example, requests to create virtual block devices using different protocols can be processed simultaneously, increasing throughput.

[0057] Further processing involves constructing a tree-structured locking mechanism based on the metadata structure tree to achieve fine-grained lock management. Each node can lock independently, preventing the entire module from being locked and thus improving concurrency. Furthermore, for each user RPC request (remote call request), its time point (the time the request arrives) and RPC request type (e.g., IO path request or non-IO path request) are obtained. A globally unique transaction ID (a uint64 integer) is created for the request based on the time point and request type. The transaction ID priority mechanism ensures fairness and efficiency in request processing: first-arriving requests are processed first, and IO path requests (which are latency-sensitive) receive higher priority, reducing user IO latency. The globally unique transaction ID facilitates comparing request priorities in deadlock avoidance mechanisms, such as determining whether a higher-priority request is waiting when locking. This system enables flexible handling of different types of requests, optimizing overall performance. Furthermore, by parsing the user RPC request corresponding to each transaction ID, the specific node requiring processing is determined. Then, starting from the central control node of the tree lock, all relevant root node paths are traversed downwards, and locking operations are performed level by level. Two-phase locking (from root down) avoids deadlocks because all requests are locked in the same order, reducing the possibility of circular waiting. Simultaneously, the Wait-Die mechanism, based on transaction ID priority, ensures that high-priority requests acquire locks first, preventing deadlocks and improving system stability. Through the lock queue of the central control node, the locking status of all requests is monitored, and abnormal requests (such as long-waiting requests) are promptly captured and terminated, ensuring the availability of metadata services. The fine-grained nature of the tree lock allows concurrent execution of requests on different subtrees, improving the concurrency and throughput of metadata services. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the main process of a method for controlling the concurrency of metadata requests in distributed block storage;

[0059] Figure 2 This is a schematic diagram simulating a sub-structure tree in a metadata request concurrency control method for distributed block storage.

[0060] Figure 3 This is a schematic diagram simulating the metadata structure tree in a method for concurrent control of metadata requests in distributed block storage.

[0061] Figure 4 This diagram illustrates the types and meanings of locks in a method for concurrent metadata request control in distributed block storage.

[0062] Figure 5 This diagram illustrates the metadata locking process in a method for concurrent metadata request control in distributed block storage.

[0063] Figure 6 This is a schematic diagram of the node read lock acquisition process in a metadata request concurrency control method for distributed block storage;

[0064] Figure 7 This is a schematic diagram of the node write lock acquisition process in a metadata request concurrency control method for distributed block storage;

[0065] Figure 8 This is a schematic diagram of the node unlocking process in a metadata request concurrency control method for distributed block storage;

[0066] Figure 9 This is a schematic diagram of the overall architecture of a concurrent metadata request control system for distributed block storage.

[0067] Figure 10 Example diagram of module division for a typical metadata management service in existing technology.

[0068] Figure label: Structure tree creation module 10, tree lock creation module 20, transaction ID creation module 30, locking processing module 40. Detailed Implementation

[0069] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings.

[0071] Example 1

[0072] like Figure 1 As shown, Embodiment 1 of the present invention provides a method for concurrent control of metadata requests for distributed block storage, including the following steps:

[0073] Step S10: Obtain the metadata of the distributed block storage system and the attribute information and business logic information corresponding to each metadata; group the metadata according to the attribute information and logic information of each metadata to establish multiple sub-tree structures; establish a common central control node for all sub-tree structures according to the business logic information between the root nodes of each sub-tree structure to obtain the metadata tree structure.

[0074] It should be noted that, in the above embodiments of this application, since the metadata service of the current distributed block storage system essentially handles the creation, deletion, modification, and querying of data objects, and the system metadata can be abstracted into a tree structure, such as... Figure 2 As shown; in Figure 2 In this embodiment, the metadata of the distributed block storage system is represented by a tree diagram (i.e., the sub-level structure tree mentioned above). For example, the Common tree is an internal data object abstraction tree; the Pool level represents a virtual storage pool, which represents a set of virtual storage objects with the same attributes / logical correlations, and it corresponds one-to-one with iSCSITarget or NVMFSubsystem; the Volume level represents user-facing storage objects, and each Volume corresponds to iSCSILun or NVMFNamespace; the Snapshot level represents a snapshot taken by a user on a virtual disk; other metadata management is self-explanatory and will not be described in detail.

[0075] This application transforms the granularity of concurrency control from "modules" to each data object in a tree structure, and, as Figure 3 As shown in the embodiment of this application, a common central control node is established for all child tree structures, and the process of establishing a common central control node for all child tree structures is described. Figure 2 Multiple trees (i.e., multiple child structure trees) are merged into a single main tree (i.e., the metadata structure tree mentioned above). Each node in the metadata structure tree is a logical abstraction or metadata entity. The difference between the child structure tree and the metadata structure tree is the addition of a control node. The child nodes of the control node are the root nodes of the atomic structure trees. The child structure tree and the metadata structure tree have a fixed hierarchical relationship, which is related to the business logic of each piece of metadata. For example:

[0076] 1. Common protocol - Pool – Volume – Snapshot;

[0077] 2. iSCSI protocol – Target – LUN;

[0078] 3. NVMf Protocol – Subsystem – NS;

[0079] 4. The overall topology – availability domain – rack – servers, etc. will not be elaborated further.

[0080] Step S20: Build a tree lock based on the metadata structure tree;

[0081] Each node in the tree lock includes a read-write lock, a set of transaction IDs currently queued for read, and a set of transaction IDs currently queued for write;

[0082] It should be noted that the above-described embodiments of this application construct a tree lock based on the metadata structure tree established according to the business logic information of each metadata. When the user's request (i.e., RPC request) arrives, the tree lock is used to lock, read, and write the nodes in the metadata structure tree according to the user's request and the above-described tree lock.

[0083] Step S30: Obtain the user's RPC request, the time point of the user's RPC request, and the RPC request type; create a transaction ID for the user's RPC request based on the time point and RPC request type;

[0084] It should be noted that in the above embodiments of this application, the aforementioned user RPC request refers to a remote call request, that is, a request from one node to another node for a service. A transaction ID (also known as a priority ID) is created for the user RPC request based on its time point and type. Specifically, user RPC requests with earlier timestamps are assigned a smaller transaction ID (e.g., 1) and thus have higher priority; user RPC requests with later timestamps are assigned a larger transaction ID (e.g., 3) and thus have lower priority than transaction ID 1. Furthermore, different RPC request types have different transaction IDs. For example, for RPC requests on the IO path, the 63rd bit of the uint64_t can be 0, while for requests on non-IO paths, the 63rd bit of the uint64_t can be 1 (which has lower priority than transaction ID 1 for RPC requests on the IO path). The first 63 bits of the uint64_t can be assigned in the manner of transaction ID 1.

[0085] In the embodiments described above, the transaction ID is specified by the metadata service when it receives an RPC request. This ID can be a uint64 integer. The larger the transaction ID, the lower the priority of the corresponding request. When locking, if there is a higher priority, it is easier to acquire the lock so that subsequent request processing can be performed earlier. The metadata service can determine the transaction ID of the corresponding request based on the priority of the request itself. For example, for requests related to logical disk data block mapping on the IO path, a lower transaction ID can be assigned to ensure that its request priority is the highest.

[0086] Step S40: Parse the user RPC requests corresponding to each transaction ID to obtain the processing object node of each user RPC request; traverse all root nodes corresponding to the processing object node, starting from the master control node of the tree lock and proceeding downwards level by level to perform locking processing by combining the transaction ID of the user RPC request and the deadlock avoidance processing operation.

[0087] It should be noted that in the above embodiments of this application, when a node in the tree lock is requested to be locked, when locking a node in the tree, it is necessary to add S locks level by level down from the root of the tree (i.e., the master control node), and then add the corresponding lock (S lock or X lock) to the node that needs to be locked. The meaning and locking type can be determined according to... Figure 4 Explanation, in Figure 4 In this context, S stands for Shared, meaning a read lock (shared lock); X stands for Exclusive, meaning a write lock (exclusive lock).

[0088] Before attempting to lock each node, a corresponding S (read) lock is first acquired on the parent node. This ensures that the parent node will not be modified during the current request processing.

[0089] Furthermore, the meaning of locking on this tree structure may also need to be clarified. When locking a node:

[0090] 1. Read lock (S lock):

[0091] Meaning: Shared access allows multiple read operations to run concurrently.

[0092] Locked range:

[0093] Current node: Read operations are allowed, but writing, renaming, removing, adding / deleting child nodes are not allowed.

[0094] Other nodes in the subtree rooted at this node (excluding the node itself): have no operation permissions (they cannot read, write, or operate on child nodes).

[0095] Note: There is no lock protection for the nodes of the subtree here, so operations on the subtree need to be locked separately.

[0096] 2. Write lock (X lock):

[0097] Meaning: Exclusive access ensures that only one operation can modify the data.

[0098] Locked range:

[0099] Current node: Allows all operations (read, write, rename, remove).

[0100] Other nodes in the subtree rooted at this node (excluding the node itself): All operations are allowed (read, write, rename, add / remove child nodes).

[0101] Note: The write lock here locks not only the current node but also the entire subtree (i.e., all descendant nodes under this node). However, note that the meaning of this locking scope is "allowing operations," not automatically holding the lock for every node in the subtree. When actually operating on a subtree node, locking still needs to be done according to the locking protocol. However, because the write lock of the parent node is held, it can be guaranteed that there will be no interference from other concurrent operations when operating on the subtree node (because other operations need to acquire the read lock of the parent node before operating on the subtree node, and the parent node is exclusively held by the write lock, so other operations will be blocked).

[0102] Furthermore, since a single metadata request involves a large number of data objects, and the tree structure may change during the request processing (because metadata service requests essentially involve adding, deleting, modifying, and querying currently managed metadata, so almost all metadata change requests will cause changes to the tree structure, for example:

[0103] 1. Create / Delete Node: Create / Delete Volume / Snapshot;

[0104] 2. Moving nodes:

[0105] a)Move Volume From Pool1 to Pool2

[0106] b)Move Lun From Target1 to Target2

[0107] c) Convert the Lun of Target1 to the NS of Subsystem1.

[0108] To prevent request deadlocks during metadata service processing, a deadlock avoidance wait-die mechanism is introduced. Therefore, this application assigns a unique transaction ID to each request received by the metadata service. This transaction ID is globally unique throughout the service's runtime, and its priority can be defined based on request type and arrival time. Before processing a request, the service needs to mark all data requiring locking and the locking type in the lock tree structure based on the data content to be accessed or modified. Then, it attempts to acquire the lock according to the relevant order and the aforementioned locking convention. When locking each involved node, if a higher-priority request is found to already hold or be queued for the corresponding data read / write lock (i.e., a lower-priority transaction ID exists among all user RPC request transaction IDs in the waiting queue), the request will stop attempting to acquire the lock, return failure directly, and be placed in the retry queue to wait for the next retry. For details on priority comparison, please refer to subsequent steps S61-S67 or S71-S77.

[0109] The embodiments described above, by reconstructing the organization of distributed block storage metadata, transform the corresponding metadata into a easily controllable tree structure and shifting the lock granularity from code modules to each data node in the tree structure, can improve the concurrency of metadata services. For example, when the metadata service receives requests to create virtual block devices with different access protocols sequentially, it can process them concurrently, improving the performance and throughput of the metadata service RPC; and by introducing a globally unique transaction ID for each request, fairness of requests of the same category is guaranteed. Transaction IDs can be maintained according to the order in which requests arrive, with first-come, first-served; the priority of different categories of requests is quantified to ensure that metadata requests on the user IO link have higher priority transaction IDs as much as possible.

[0110] Meanwhile, by introducing the concept of two-phase locking, deadlocks caused by arbitrary locking during request processing are avoided. For requests that update multiple metadata entities in batches, transaction IDs are used to avoid deadlocks during concurrency, so as to ensure that there is no circular waiting due to resource requests during the operation and to avoid metadata service unavailability.

[0111] In addition, since all processed RPC requests need to acquire a read or write lock on the master node, the availability of metadata services can be guaranteed by periodically checking the transaction ID set of the master node's read and write queues, promptly capturing abnormal request IDs and terminating them.

[0112] The embodiments described above, based on the hierarchical nature of distributed storage metadata, organize metadata resources into a dynamic tree structure according to business topology, achieving strict matching between lock granularity and data / control topology. When different requests involve different subtree operations, the requests can be executed in complete parallel, improving service concurrency; and through a lightweight transactional deadlock avoidance mechanism, the request priority is combined with the request transaction ID, which can minimize request latency on the IO path, ensure fairness for requests of equal priority, and avoid service deadlock.

[0113] The embodiments described above first acquire all metadata of the distributed block storage system, including attribute information (such as type, location, etc.) and business logic information (such as association relationships, access modes, etc.) for each metadata. Then, the metadata is grouped according to this information, with each group forming a sub-tree structure (e.g., Common tree, iSCSI tree, NVM tree, etc.) to reflect the internal logical hierarchy of the metadata. Finally, based on the business logic information (such as protocol associations, topology relationships) between the root nodes of each sub-tree structure, a common central control node is established for all subtrees, thereby integrating them into a complete metadata structure tree. This transforms metadata from coarse-grained management at the module level to fine-grained management at the object level. The tree structure intuitively represents the business logic relationships of the metadata, and the central control node provides a global view, enabling subsequent concurrency control to be based on the entire tree structure, rather than processing each module in isolation. This lays the foundation for the construction of a tree lock. At the same time, the tree structure allows requests from different subtrees to be processed in parallel, thereby improving the concurrency performance of the metadata service; for example, requests to create virtual block devices with different protocols can be processed simultaneously, improving throughput.

[0114] Further processing involves constructing a tree-structured locking mechanism based on the metadata structure tree to achieve fine-grained lock management. Each node can lock independently, preventing the entire module from being locked and thus improving concurrency. Furthermore, for each user RPC request (remote call request), its time point (the time the request arrives) and RPC request type (e.g., IO path request or non-IO path request) are obtained. A globally unique transaction ID (a uint64 integer) is created for the request based on the time point and request type. The transaction ID priority mechanism ensures fairness and efficiency in request processing: first-arriving requests are processed first, and IO path requests (which are latency-sensitive) receive higher priority, reducing user IO latency. The globally unique transaction ID facilitates comparing request priorities in deadlock avoidance mechanisms, such as determining whether a higher-priority request is waiting when locking. This system enables flexible handling of different types of requests, optimizing overall performance. Furthermore, by parsing the user RPC request corresponding to each transaction ID, the specific node requiring processing is determined. Then, starting from the central control node of the tree lock, all relevant root node paths are traversed downwards, and locking operations are performed level by level. Two-phase locking (from root down) avoids deadlocks because all requests are locked in the same order, reducing the possibility of circular waiting. Simultaneously, the Wait-Die mechanism, based on transaction ID priority, ensures that high-priority requests acquire locks first, preventing deadlocks and improving system stability. Through the lock queue of the central control node, the locking status of all requests is monitored, and abnormal requests (such as long-waiting requests) are promptly captured and terminated, ensuring the availability of metadata services. The fine-grained nature of the tree lock allows concurrent execution of requests on different subtrees, improving the concurrency and throughput of metadata services.

[0115] Specifically, in step S40, the process of traversing all root nodes corresponding to the processing object node, starting from the master control node of the tree lock and proceeding downwards level by level, combines the transaction ID of the user's RPC request and deadlock avoidance operations to perform locking. This process also includes the following steps:

[0116] Step S50: Based on the data memory snapshot corresponding to the transaction ID of the current user RPC request, the processing object node repeatedly runs the current user RPC request for a preset time period of waiting time. After the current user RPC request is completed, the corresponding processing object node is unlocked.

[0117] It should be noted that in the above embodiments of this application, the resilience, consistency and efficiency of the distributed block storage metadata service are improved through the retry mechanism and data memory snapshot. For details of the operation, please refer to the following steps S51-S54.

[0118] Specifically, such as Figure 5 As shown, in step S50, based on the data memory snapshot corresponding to the transaction ID of the current user's RPC request, the processing object node repeatedly runs the current user's RPC request after a preset waiting period. After the current user's RPC request is completed, the corresponding processing object node is unlocked, including the following operation steps:

[0119] Step S51: Obtain the transaction ID of the current user's RPC request;

[0120] Step S52: Obtain a data memory snapshot of the current RPC request based on the transaction ID (the data memory snapshot includes metadata attribute information and request semantic information); perform locking on the tree lock based on the data memory snapshot;

[0121] The locking process includes read lock acquisition and write lock acquisition;

[0122] Step S53: Determine whether the locking was successful; if yes, verify whether the locking process meets the expected result based on the data memory snapshot; if yes (if no, return to the locking process of step S52 above after a preset time period and re-execute until the unlocking process is implemented.), then process the current user's RPC request, and after processing the current user's RPC request, implement the unlocking process on the processing object node;

[0123] Step S54: If locking fails, return to step S52 above and re-execute the locking process after a preset time period until the node to be processed is unlocked.

[0124] It should be noted that in the above embodiments of this application, step S51 is the starting point of the retry loop. When a request enters the retry process due to locking failure, the system first needs to determine which request is currently being processed. The transaction ID is a globally unique identifier assigned to each user RPC request in step S30. It not only identifies the request itself, but more importantly, its value represents the priority of the request. Through the implementation of step S51, the identity and priority of the request are confirmed, ensuring that throughout the retry process, the system always executes the locking strategy and deadlock avoidance judgment (Wait-Die mechanism) based on the priority (transaction ID) of this specific request, thereby maintaining the consistency and fairness of priority scheduling. Furthermore, in step S52, based on the transaction ID obtained in step S51, a snapshot of the metadata status related to the request is collected. This snapshot not only includes the current ownership of the metadata node to be operated on by the request, but also... The request includes not only metadata information (such as size and status) but also semantic information (i.e., the operation the request intends to perform, such as "create volume" or "delete snapshot"). Then, based on the information contained in the data memory snapshot (i.e., which nodes need to be accessed and what operations need to be performed), the locking process is re-initiated to ensure the consistency of the data view during retries and to safely attempt to acquire resources. Furthermore, in a distributed high-concurrency environment, the metadata state may have been modified by other requests between two retries. By taking a new snapshot for each retry attempt, it can be ensured that the data state used in this retry cycle is deterministic and consistent, avoiding decisions based on expired or dirty data, which is key to ensuring the correctness of the operation. At the same time, by re-traversing the tree lock structure based on the latest snapshot information, it is ensured that the locking request matches the latest metadata topology, thereby safely and correctly attempting to acquire the required lock resources.Furthermore, in step S53, it is checked whether all necessary locks have been successfully acquired as intended in step S52. That is, even if the locking is successful in the program logic, the system still needs to verify whether the current locked state is consistent with the expected state of the snapshot collected in step S52. For example, during the locking process, whether the target node has been deleted or moved, or whether its key attributes have changed. Verification ensures that the assumptions of the previous snapshot still hold after the lock is acquired. If verification is successful, it indicates that the data state has not changed unexpectedly, and the specific business logic of the request (such as modifying metadata) can be safely executed. After processing, the locks held by the request on all processing object nodes are released (unlocked). If verification fails, it indicates that the data state has changed unexpectedly within the very short time window of locking, rendering the locking result invalid. In this case, the system will not execute the request. Instead of retrying the business logic, the system releases the acquired lock, waits for a preset time period, and then returns to step S52 to start a new round of attempts (re-collecting snapshots and re-locking). The system passes the consistency check in step S53 and completes the request or triggers a retry, enhancing the robustness of the protocol and preventing dangerous operations from being performed when the data state is invalid. Only after successful verification will the request be truly executed and the lock finally released, signifying that the request has been successfully processed. The retry mechanism after verification failure effectively handles extreme competition conditions under high concurrency, ensuring that the system can always reach a consistent state through retries. However, if the initial locking fails in step S53 (for example, when attempting to lock a node, the Wait-Die mechanism detects the existence of a higher-priority request and actively abandons the attempt), the request will not retry immediately but will enter a waiting state. After waiting for a preset time period, the system jumps back to step S52 to start a new attempt, achieving negative feedback control and avoiding livelock and system overload. This is because if failed requests are retried immediately without delay, multiple requests may continuously conflict and repeatedly fail, forming a "livelock" situation. The preset time period waiting mechanism set in this embodiment breaks this immediate competition. Furthermore, by allowing failed requests to wait for a period of time, the retry frequency is reduced, alleviating the continuous pressure on the metadata service (especially the lock manager) and freeing up resources for processing other requests. At the same time, the preset waiting time allows the system state (lock holding status) time to change, thereby increasing the probability of successful retry and improving the overall throughput and efficiency of the system.

[0125] Specifically, such as Figure 6 As shown, the read lock acquisition process includes the following steps:

[0126] Step S61: Traverse all nodes in the tree lock and determine whether the current node is the root node;

[0127] Step S62: If the current node is determined to be the root node, then further determine whether there are any user RPC requests other than the current write request in the processing queue of the current node; if it is determined that there are no user RPC requests other than the current write request in the processing queue of the current node, then put the current transaction ID into the set of transaction IDs in the queue of read requests, call the read blocking processing of the read-write lock, and output the lock acquisition success result.

[0128] Step S63: If it is determined that there are user RPC requests other than the current write request in the processing queue of the current node, then further determine whether the transaction ID of the current user RPC request is less than the transaction ID of user RPC requests other than the current write request in the write request processing queue; if so, put the current transaction ID into the transaction ID set of the read request queue, call the read blocking processing of the read-write lock, and output the lock acquisition success result.

[0129] Step S64: If it is determined that the transaction ID of the current user RPC request is not less than the transaction ID of the user RPC requests in the write request processing queue other than the current write request, then determine whether a read lock has been acquired on the parent node of the current root node.

[0130] Step S65: If it is determined that a read lock has been acquired on the parent node of the current root node, then release the read lock on the parent node of the current root node and return the result of lock acquisition failure; if it is determined that a read lock has not been acquired on the parent node of the current root node, then directly return the result of lock acquisition failure.

[0131] Step S66: If it is determined that the current node is not the root node, then acquire a read lock on the parent node of the current node; check whether the read lock acquisition on the parent node of the current node was successful.

[0132] Step S67: If it is determined that the parent node of the current node has successfully acquired the read lock, return to the above processing queue for judgment until the output results in either lock acquisition failure or lock acquisition success.

[0133] It should be noted that, as explained above, the embodiments of this application described above are deadlock avoidance operations and priority comparison processing of user RPC requests. To ensure that request deadlocks do not occur during the processing of the metadata service, a deadlock avoidance Wait-Die mechanism needs to be introduced. Therefore, this application assigns a request transaction ID to each request received by the metadata service. This ID is globally unique throughout the service's runtime, and its priority can be defined based on the request type and arrival time during operation. Before processing a request, the service needs to mark all data requiring locking and the locking type in the lock tree structure based on the data content to be accessed or modified. Then, it attempts to lock according to the relevant order and the aforementioned locking convention. When locking each involved node, if a higher-priority request is found to already hold or be in the queue waiting for the corresponding data read / write lock, the request will stop attempting to lock, return failure directly, and be placed in the retry queue to wait for the next retry.

[0134] Specifically, such as Figure 7 As shown, the write lock acquisition process includes the following steps:

[0135] Step S71: Traverse all nodes in the tree lock and determine whether the current node is the root node;

[0136] Step S72: If the current node is determined to be the root node, then further determine whether there are any user RPC requests other than the current read / write request in the processing queue of the current node; if it is determined that there are no user RPC requests other than the current read / write request in the processing queue of the current node, then put the current transaction ID into the transaction ID set of the write request queue, call the write blocking handling of the read / write lock, and output the lock acquisition success result.

[0137] Step S73: If it is determined that there are user RPC requests other than the current write request in the processing queue of the current node, then further determine whether the transaction ID of the current user RPC request is less than the transaction ID of user RPC requests other than the current write request in the write request processing queue, and at the same time satisfy whether the transaction ID of the current user RPC request is less than the transaction ID of user RPC requests other than the current write request in the read request processing queue.

[0138] Step S74: If it is determined that the transaction ID of the current user's RPC request is less than the transaction ID of user RPC requests other than the current write request in the write request processing queue, and the transaction ID of the current user's RPC request is less than the transaction ID of user RPC requests other than the current write request in the read request processing queue, then it is determined whether a read lock has been acquired on the parent node of the current root node; if so, the current transaction ID is added to the set of transaction IDs in the write request queue, and the write blocking handling of the read-write lock is called, and the lock acquisition success result is output.

[0139] Step S75: If it is determined that a read lock has been acquired on the parent node of the current root node, then release the read lock on the parent node of the current root node and return the result of lock acquisition failure; if it is determined that a read lock has not been acquired on the parent node of the current root node, then directly return the result of lock acquisition failure.

[0140] Step S76: If it is determined that the current node is not the root node, then acquire a read lock on the parent node of the current node; check whether the read lock acquisition on the parent node of the current node was successful.

[0141] Step S77: If it is determined that the parent node of the current node has successfully acquired the read lock, return to the above processing queue for judgment until the output results in either lock acquisition failure or lock acquisition success.

[0142] Specifically, such as Figure 8 As shown, in step S53 or step S54, the unlocking process is performed on the target node, including the following steps:

[0143] Step S81: Determine whether the currently processed object node is the root node;

[0144] Step S82: If the current processing object node is determined to be the root node, the unlocking instruction of the read / write lock is called to remove the current transaction ID from the set of transaction IDs queued for read / write requests;

[0145] Step S83: If it is determined that the current processing object node is not the root node, then obtain the parent node of the current processing object node, call the unlocking instruction of the read / write lock, and remove the current transaction ID from the set of transaction IDs queued for read / write requests.

[0146] It should be noted that the above process of removing the current transaction ID from the set of transaction IDs queued for read / write requests achieves the unlocking of the processing object node.

[0147] Specifically, the process involves traversing all root nodes corresponding to the processing object node, starting from the master node of the tree lock and proceeding downwards level by level to perform locking processing based on the transaction ID of the user's RPC request and deadlock avoidance operations. Following this, the process also includes...

[0148] If the duration of the lock on a node in the structure tree by the transaction ID reaches a duration threshold within a preset time period, then the node in the structure tree will be unlocked.

[0149] It's important to note that since all processed metadata requests require acquiring a read or write lock on the root node (i.e., the master node) of the tree structure, a timer can be defined in the metadata service to periodically check the transaction ID set of the master node's read / write lock queue. This allows for timely detection of abnormal requests and the termination of related requests to ensure the availability of the metadata service. For example, the master node's lock queue set can be checked every 60 seconds. If the queue contained {1, 2, 3} in the previous check, and now contains {2, 100, 101}, this indicates that ID=1 has held the lock for 60 seconds, which can be considered an abnormal request and needs to be terminated.

[0150] Example 2

[0151] like Figure 9 As shown, the metadata request concurrency control system for distributed block storage provided in Embodiment 2 of the present invention includes a structure tree establishment module 10, a tree lock establishment module 20, a transaction ID creation module 30, and a locking processing module 40.

[0152] The structure tree establishment module 10 is used to group the metadata in the distributed block storage system according to the attribute information and business logic information corresponding to each metadata in the distributed block storage system to obtain multiple sub-structure trees; and to establish a common master control node for all sub-tree structures according to the business logic information between the root nodes of each sub-tree structure to obtain the metadata structure tree.

[0153] The tree lock establishment module 20 is used to establish a tree lock based on the metadata structure tree;

[0154] Each node in the tree lock includes a read-write lock, a set of transaction IDs currently queued for read, and a set of transaction IDs currently queued for write;

[0155] The transaction ID creation module 30 is used to obtain the user's RPC request, the time point of the user's RPC request, and the type of the RPC request; and to create a transaction ID for the user's RPC request based on the time point and type of the user's RPC request.

[0156] The locking processing module 40 is used to parse the user RPC requests corresponding to each transaction ID to obtain the processing object node of each user RPC request; based on all root nodes corresponding to the processing object node, it traverses and sequentially performs locking processing by combining the transaction ID of the user RPC request and deadlock avoidance processing operations from the master control node of the tree lock downwards.

[0157] In summary, the present invention provides a method and system for concurrent metadata request control in distributed block storage. This method acquires all metadata from the distributed block storage system, including attribute information (such as type and location) and business logic information (such as associations and access patterns) for each metadata element. Then, based on this information, the metadata is grouped, with each group forming a sub-tree structure (e.g., Common tree, iSCSI tree, NVM tree), reflecting the internal logical hierarchy of the metadata. Finally, based on the business logic information (such as protocol associations and topology relationships) between the root nodes of each sub-tree structure, the method further controls the metadata's internal logical hierarchy. This approach establishes a common central control node for all subtrees, integrating them into a complete metadata structure tree. This transforms metadata management from coarse-grained (module-level) to fine-grained (object-level) management. The tree structure visually represents the business logic relationships of metadata, and the central control node provides a global view, enabling subsequent concurrency control to be based on the entire tree structure rather than processing each module in isolation. This lays the foundation for building a tree-based lock. Furthermore, the tree structure allows for parallel processing of requests from different subtrees, improving the concurrency performance of metadata services; for example, requests to create virtual block devices using different protocols can be processed simultaneously, increasing throughput.

[0158] Further processing involves constructing a tree-structured locking mechanism based on the metadata structure tree to achieve fine-grained lock management. Each node can lock independently, preventing the entire module from being locked and thus improving concurrency. Furthermore, for each user RPC request (remote call request), its time point (the time the request arrives) and RPC request type (e.g., IO path request or non-IO path request) are obtained. A globally unique transaction ID (a uint64 integer) is created for the request based on the time point and request type. The transaction ID priority mechanism ensures fairness and efficiency in request processing: first-arriving requests are processed first, and IO path requests (which are latency-sensitive) receive higher priority, reducing user IO latency. The globally unique transaction ID facilitates comparing request priorities in deadlock avoidance mechanisms, such as determining whether a higher-priority request is waiting when locking. This system enables flexible handling of different types of requests, optimizing overall performance. Furthermore, by parsing the user RPC request corresponding to each transaction ID, the specific node requiring processing is determined. Then, starting from the central control node of the tree lock, all relevant root node paths are traversed downwards, and locking operations are performed level by level. Two-phase locking (from root down) avoids deadlock because all requests are locked in the same order, reducing the possibility of circular waiting. Simultaneously, the Wait-Die mechanism, based on transaction ID priority, ensures that high-priority requests acquire locks first, preventing deadlock and improving system stability. Through the lock queue of the central control node, the locking status of all requests is monitored, and abnormal requests (such as long-waiting requests) are promptly captured and terminated, ensuring the availability of metadata services. The fine-grained nature of tree locks allows concurrent execution of requests on different subtrees, improving the concurrency and throughput of metadata services.

[0159] Furthermore, based on the hierarchical nature of distributed storage metadata, metadata resources are organized into a dynamic tree structure according to business topology, achieving strict matching between lock granularity and data / control topology. When different requests involve different subtree operations, the requests can be executed in complete parallel, improving service concurrency.

[0160] Meanwhile, a lightweight transactional deadlock avoidance mechanism is applied, which combines request priority with request transaction ID. This can minimize request latency on the IO path, ensure fairness for requests of equal priority, and prevent service deadlock.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; those skilled in the art can modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for concurrent control of metadata requests for distributed block storage, characterized in that, The following steps are included: Based on the attribute information and business logic information corresponding to each metadata in the distributed block storage system, the metadata in the distributed block storage system is grouped to establish multiple sub-level tree structures; a common central control node is established for all sub-level tree structures based on the business logic information between the root nodes of each sub-level tree structure to obtain the metadata tree structure. A tree lock is obtained by constructing a tree structure based on the metadata structure tree; Obtain the user's RPC request, including the time and type of the RPC request; Create a transaction ID for the user's RPC request based on the time and type of the RPC request; Parse the user RPC requests corresponding to each transaction ID to obtain the processing object node for each user RPC request; The process involves traversing all root nodes corresponding to the processing object node, starting from the master node of the tree lock and sequentially performing locking operations downwards. This process also includes the following steps: obtaining the transaction ID of the current user's RPC request; acquiring a data memory snapshot of the current RPC request based on the transaction ID; performing locking operations on the tree lock based on the data memory snapshot; the locking operation includes acquiring read locks and acquiring write locks; determining whether the locking was successful; if so, verifying whether the locking operation meets the expected result based on the data memory snapshot; if so, processing the current user's RPC request, and unlocking the processing object node after processing the current user's RPC request. The read lock acquisition process includes the following steps: Traverse all nodes in the tree lock and determine whether the current node is the root node; If the current node is determined to be the root node, then it is further determined whether there are any user RPC requests other than the current write request in the processing queue of the current node; If it is determined that there are no user RPC requests other than the current write request in the current node's processing queue, then the current transaction ID is added to the set of transaction IDs in the read request queue, and the read blocking handling of the read-write lock is invoked, outputting the result of successful locking; If it is determined that there are user RPC requests other than the current write request in the current node's processing queue, then it is further determined whether the transaction ID of the current user RPC request is less than the transaction ID of user RPC requests other than the current write request in the write request processing queue. If so, add the current transaction ID to the set of transaction IDs in the read request queue, call the read blocking handling of the read-write lock, and output the result of successful locking; If it is determined that the transaction ID of the current user's RPC request is not less than the transaction ID of the user's RPC requests in the write request processing queue other than the current write request, then determine whether a read lock has been acquired on the parent node of the current root node. If it is determined that a read lock has been acquired on the parent node of the current root node, then the read lock on the parent node of the current root node is released, and the result of lock acquisition failure is returned. If it is determined that a read lock has not been acquired on the parent node of the current root node, then the lock acquisition failure result is returned directly; If it is determined that the current node is not the root node, then a read lock is acquired on the parent node of the current node; Determine whether the parent node of the current node has successfully acquired a read lock.

2. The method for concurrent control of metadata requests for distributed block storage according to claim 1, characterized in that, Each node in the tree lock includes a read-write lock, a set of transaction IDs currently queued for read, and a set of transaction IDs currently queued for write.

3. The method for concurrent control of metadata requests for distributed block storage according to claim 2, characterized in that, The data memory snapshot includes metadata attribute information and request semantic information.

4. The method for concurrent control of metadata requests for distributed block storage according to claim 3, characterized in that, The write lock acquisition process includes the following steps: Traverse all nodes in the tree lock and determine whether the current node is the root node; If the current node is determined to be the root node, then it is further determined whether there are any user RPC requests other than the current read / write request in the processing queue of the current node; If it is determined that there are no user RPC requests other than the current read / write request in the current node's processing queue, then the current transaction ID is added to the set of transaction IDs in the write request queue, and the write blocking handling of the read / write lock is invoked, outputting the result of successful locking; If it is determined that there are user RPC requests other than the current write request in the current node's processing queue, then it is further determined whether the transaction ID of the current user RPC request is less than the transaction ID of user RPC requests other than the current write request in the write request processing queue, and at the same time, it is also determined whether the transaction ID of the current user RPC request is less than the transaction ID of user RPC requests other than the current write request in the read request processing queue. If it is determined that the transaction ID of the current user's RPC request is less than the transaction ID of the user's RPC request in the write request processing queue (excluding the current write request) and the transaction ID of the current user's RPC request is less than the transaction ID of the user's RPC request in the read request processing queue (excluding the current write request), then it is determined whether a read lock has been acquired on the parent node of the current root node. If so, add the current transaction ID to the set of transaction IDs in the write request queue, call the write blocking handling of the read-write lock, and output the lock acquisition success result; If it is determined that a read lock has been acquired on the parent node of the current root node, then the read lock on the parent node of the current root node is released, and the result of lock acquisition failure is returned. If it is determined that a read lock has not been acquired on the parent node of the current root node, then the lock acquisition failure result is returned directly; If it is determined that the current node is not the root node, then a read lock is acquired on the parent node of the current node; Determine whether the parent node of the current node has successfully acquired a read lock.

5. A method for concurrent control of metadata requests for distributed block storage according to claim 4, characterized in that, The unlocking process for the target node includes the following steps: Determine if the currently processed object node is the root node; If the node being processed is determined to be the root node, the unlocking instruction for the read / write lock is invoked, and the current transaction ID is removed from the set of transaction IDs queued for read / write requests. If it is determined that the currently processed object node is not the root node, then obtain the parent node of the currently processed object node, call the unlocking instruction of the read / write lock, and remove the current transaction ID from the set of transaction IDs queued for read / write requests.

6. A method for concurrent control of metadata requests for distributed block storage according to claim 5, characterized in that, The process involves traversing all root nodes corresponding to the processing object node, starting from the master node of the tree lock and sequentially performing locking operations downwards. This also includes... If the duration of the lock on a node in the structure tree by the transaction ID reaches a duration threshold within a preset time period, then the node in the structure tree will be unlocked.

Images