Distributed lease management method and system, storage medium and electronic device

Abstract

This application discloses a distributed lease management method and system, storage medium, and electronic device, relating to the field of storage management technology. It is used at the client layer and includes: upon receiving a file operation request, sending a lease request to a node layer, wherein the lease request indicates a target lease for the target file corresponding to the file operation request; after receiving the target lease sent by the node layer, accessing the target file in the storage system according to the target lease, wherein the node layer corresponds to at least one client layer; in the event of a conflict between the target lease and a reference node layer, the node layer accesses the cluster layer; and upon receiving an authorization instruction sent by the cluster layer, the node layer sends the target lease. This solves the technical problem of low resource utilization in distributed file systems in related technologies, achieving the technical effect of improving file system resource utilization.

Description

Technical Field

[0001] This application relates to the field of storage management technology, and in particular to a distributed lease management method and system, storage medium and electronic device. Background Technology

[0002] The demand for high-concurrency file access in multi-client, multi-tenant scenarios is growing. In these scenarios, multiple containerized applications, microservices, or analytics engines may need to read and write the same or related file sets simultaneously to process critical business data such as transaction logs, log files, and model training data.

[0003] In related technologies, most distributed file systems employ a centralized lock management strategy, where a central node coordinates file lock requests from all clients. This model works in small-scale systems, but in large-scale, multi-client environments, the central node becomes a severe bottleneck. Frequent centralized lock requests and releases lead to network I / O storms, significantly increasing system latency and reducing throughput. In other words, distributed file systems in related technologies suffer from low resource utilization. Summary of the Invention

[0004] This application provides a distributed lease management method and system, storage medium and electronic device to at least solve the problem of low resource utilization in distributed file systems in related technologies.

[0005] This application provides a distributed lease management method for a client layer, comprising: upon receiving a file operation request, sending a lease request to a node layer, wherein the lease request indicates a target lease for a target file corresponding to the file operation request; upon receiving the target lease sent by the node layer, accessing the target file in the storage system according to the target lease, wherein the node layer corresponds to at least one client layer, and in the event of a conflict between the target lease and a reference node layer, the node layer accesses the cluster layer; and upon receiving an authorization instruction sent by the cluster layer, the node layer sends the target lease.

[0006] This application also provides a distributed lease management device for a client layer, comprising: a request sending module, configured to send a lease request to a node layer upon receiving a file operation request, wherein the lease request indicates a target lease for a target file corresponding to the file operation request; and an access module, configured to access a target file in a storage system according to the target lease after receiving the target lease sent by the node layer, wherein the node layer corresponds to at least one client layer, and in the event of a conflict between the target lease and a reference node layer, the node layer accesses the cluster layer, and in the event of an authorization instruction sent by the cluster layer, the node layer sends the target lease.

[0007] This application provides another distributed lease management method for the node layer, comprising: receiving a lease request sent by the client layer, wherein the lease request is used to indicate a target lease for a target file corresponding to a file operation request; querying a local lease table, and if the target lease is held by a reference node layer, sending a query request to the cluster layer, wherein the query request is used to query the status of the target lease; and sending the target lease to the client layer upon receiving an authorization instruction sent by the cluster layer.

[0008] This application also provides another distributed lease management device for the node layer, comprising: a request receiving module for receiving lease requests sent by the client layer, wherein the lease request indicates a target lease for a target file corresponding to a file operation request; a lease table query module for querying a local lease table and, if the target lease is held by a reference node layer, sending a query request to the cluster layer, wherein the query request is used to query the status of the target lease; and a lease sending module for sending the target lease to the client layer upon receiving an authorization instruction sent by the cluster layer.

[0009] This application provides yet another distributed lease management method for a cluster layer, comprising: receiving a query request sent by a node layer, wherein the query request is used to query the status of a target lease, and the target lease is the lease of a target file corresponding to a file operation request processed by the client layer; determining a global lease index based on the query request, wherein the global lease index is used to indicate the local lease tables corresponding to the node layer and at least one reference node layer; and sending an authorization instruction to the node layer if the reference node layer releases the target lease, wherein the authorization instruction is used to indicate that the node layer is allowed to send the target lease to the client layer if it receives the authorization instruction sent by the cluster layer.

[0010] This application also provides another distributed lease management device for a cluster layer, comprising: a query receiving module for receiving query requests sent by a node layer, wherein the query request is used to query the status of a target lease, and the target lease is the lease of a target file corresponding to a file operation request processed by a client layer; an index determining module for determining a global lease index based on the query request, wherein the global lease index is used to indicate the local lease tables corresponding to the node layer and at least one reference node layer; and an authorization module for sending an authorization instruction to the node layer when a reference node layer releases the target lease, wherein the authorization instruction is used to indicate that the node layer is allowed to send the target lease to the client layer when it receives the authorization instruction sent by the cluster layer.

[0011] This application also provides a distributed lease management system, comprising: a client layer for executing any of the distributed lease management methods described above; a node layer for executing any of the distributed lease management methods described above; a cluster layer for executing any of the distributed lease management methods described above; and a storage device for storing multiple files, including a target file.

[0012] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described distributed lease management methods.

[0013] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described distributed lease management methods.

[0014] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described distributed lease management methods.

[0015] This application describes a system where, upon receiving a file operation request, the client layer sends a lease request to the node layer. This lease request indicates the target lease for the target file corresponding to the file operation request. After receiving the target lease from the node layer, the client layer accesses the target file in the storage system based on the target lease. Each node layer corresponds to at least one client layer. In the event of a conflict between the target lease and a reference node layer, the node layer accesses the cluster layer. By delegating lease request processing from the remote cluster layer to the local node layer, communication latency is significantly reduced, and the response speed of file operations is improved. When the client layer receives a file operation request, it intercepts the POSIX API and manages the lease request locally, avoiding the overhead of direct network round trips to the remote MDS or Monitor for each file operation. This results in more efficient overall system performance, especially for high-frequency read / write operations, significantly reducing latency. Once the client layer successfully obtains the target lease from the node layer, it can directly access the target file through the distributed storage system without further lock waiting or queries. This immediate access capability improves the efficiency of file operations, particularly for read operations, as acquiring read leases typically does not cause conflicts and can be completed quickly. For write operations, the existence of leases ensures exclusivity, thus avoiding data inconsistency and guaranteeing data integrity and security. Therefore, it can solve the problem of low resource utilization in distributed file systems of related technologies. Attached Figure Description

[0016] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the hardware environment of an optional distributed lease management method according to an embodiment of this application;

[0018] Figure 2 This is a flowchart of an optional distributed lease management method according to an embodiment of this application;

[0019] Figure 3 This is a flowchart of another optional distributed lease management method according to an embodiment of this application;

[0020] Figure 4 This is a flowchart of another optional distributed lease management method according to an embodiment of this application;

[0021] Figure 5 This is a structural block diagram of an optional distributed lease management device according to an embodiment of this application;

[0022] Figure 6 This is a structural block diagram of another optional distributed lease management device according to an embodiment of this application;

[0023] Figure 7 This is a structural block diagram of another optional distributed lease management device according to an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0025] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0026] It's important to note that a lock lease is a mechanism used in distributed systems to manage access permissions to shared resources (such as files and database records). In traditional locking mechanisms, resource locks are typically managed by a central server. In contrast, a lock lease mechanism allows multiple clients to hold control of a resource for a specific period, known as the lease's validity period. The client holding the lock lease can perform read and write operations on the resource during the lease's term, while other clients must wait or be blocked until they acquire a corresponding lock lease.

[0027] In a distributed environment, lock lease mechanisms can reduce the burden on the central server, improve system concurrency and performance, and ensure data consistency and integrity. For example, in a distributed file system, when a client requests a read or write lock on a file, the system decides whether to grant the lock based on the rules of the lock lease and guarantees access control to the resource during the lock's validity period.

[0028] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] According to one aspect of the embodiments of this application, a distributed lease management method is provided. As an optional implementation, the above-described distributed lease management method can be applied to, but is not limited to, [examples of other methods]. Figure 1 The hardware environment shown depicts a distributed lease management system. This distributed lease management system may include, but is not limited to, a client layer (Client), a control plane (Monitor / MDS), and a data plane (OSD cluster).

[0030] The client layer is the first point of entry for users to interact directly with the distributed storage system. It provides a variety of storage interfaces, including block storage, object storage, and file storage interfaces, which are used to provide block device services, object gateway services, and file system services, respectively, so that users can choose the appropriate data access mode according to their actual application needs.

[0031] The librados library is a crucial component of the client layer. It encapsulates the APIs for low-level communication with distributed storage clusters, including data reading and writing, and metadata retrieval. The librados library enables clients to interact with OSDs (Object Storage Devices) in the cluster through a unified interface, without needing to concern themselves with the specific implementation details of the underlying storage.

[0032] The CRUSH client module is responsible for locally caching data according to the CRUSH algorithm and calculating the physical location of data stored in the storage pool. When a client needs to store or retrieve data, the CRUSH module performs a hash calculation based on the object's name, and then traverses the CRUSH bucket hierarchy (e.g., host → rack → data center) according to the storage pool's rules (number of replicas, fault domain level, etc.), dynamically selecting the most suitable OSD node based on the weight of each bucket. This mechanism ensures a balanced distribution of data across the cluster while satisfying redundancy and fault tolerance strategies; for example, data replicas are deployed across racks to reduce the impact of local failures.

[0033] The control plane consists of two core components: Monitor and MDS (Metadata Server), which are responsible for metadata management, state consistency, and coordination of critical services in the distributed storage system.

[0034] The Monitor component maintains cluster metadata, including OSD Maps (describing which OSD nodes are responsible for which data), CRUSH Maps (storage policies and replica distribution rules), authentication information, etc., ensuring that this metadata remains consistent across multiple Monitor nodes using the Paxos algorithm. Upon initial connection, clients pull the latest cluster metadata from the Monitor and synchronize with it periodically via a heartbeat mechanism to obtain the latest status and rules, ensuring the correctness and efficiency of data operations.

[0035] In file storage scenarios, MDS is responsible for managing file system metadata, such as directory tree structure and file permissions, and caching frequently accessed metadata to accelerate frequently accessed file read and write operations. MDS not only maintains metadata information, but also supports subtree lease management, allowing clients to directly access and operate on all files under a subtree when holding a subtree lease, without having to request a lease for each file separately, thereby improving the performance of batch file operations.

[0036] The data plane consists of multiple OSD nodes, each responsible for data storage, replication, data protection coding (such as erasure coding), and data consistency maintenance. OSD nodes perceive the entire cluster topology through an OSD Map, including node additions and departures, and changes in node states, enabling them to intelligently store and distribute data.

[0037] OSD node hardware typically includes a high-performance CPU, ample memory, high I / O disks (such as HDDs and SSDs), and high-speed network interfaces (such as gigabit or 10-gigabit network cards). This hardware configuration ensures fast data storage and access, as well as efficient data exchange between nodes. OSD nodes are responsible for storing user data, executing data replication strategies to maintain data redundancy, and applying data protection technologies such as erasure coding to improve storage efficiency and data recovery capabilities. Furthermore, OSD nodes participate in data consistency maintenance, ensuring the correct distribution and update order of data within the cluster.

[0038] Figure 1 The architecture works collaboratively, enabling distributed storage systems to support large-scale concurrent access and data storage needs. The client layer provides the user interface, the control plane is responsible for state consistency and metadata management, and the data plane focuses on data storage and protection. Connected via high-speed networks (such as Ethernet and InfiniBand), these components can efficiently exchange information and data, achieving distributed data management and storage. This decentralized and layered architecture not only improves the system's scalability and performance but also enhances data fault tolerance and security, making it ideal for applications such as cloud storage, big data analytics, and high-performance computing.

[0039] The distributed lease management method of this application can be implemented jointly by the client layer, node layer, and cluster layer, wherein the client layer, node layer, and cluster layer are... Figure 1 The position in the middle is as follows:

[0040] The client layer (LibFS-client) resides on the distributed storage client side, that is, in each client system that initiates a request to the distributed storage system. It typically exists as a user-space file system library, integrated with applications via LD_PRELOAD or other dynamic linking methods, intercepting and handling POSIX file operation APIs such as open, read, and write. The client layer is the first layer of interaction between user programs and the distributed storage system; it can locally manage process leases and communicate with the node layer to coordinate lease requests.

[0041] The node layer (DFS-server) resides on each compute node in the data plane and works in conjunction with the object storage device (OSD). It typically runs in the privileged user environment of the compute node, listening on local sockets or shared memory, receiving lease requests from the client layer, and managing the lease status of all processes within the node, handling local lease conflicts. Simultaneously, the node layer uses RDMA technology for efficient data synchronization with hot replica nodes, and coordinates cross-node lease conflicts and status updates with the cluster layer.

[0042] The cluster layer (Monitor / MDS) resides in the control plane and consists of multiple Monitor and MDS service nodes. It ensures the consistency of cluster metadata and lease status through distributed consensus algorithms such as Paxos. MDS manages file system metadata, including directory trees and permissions, and supports subtree lease mechanisms. The Monitor maintains the cluster's OSD Map, CRUSH Map, and authentication information, keeping synchronized with the client layer through a heartbeat mechanism. The cluster layer acts as the global state manager and coordinator throughout the system, responsible for maintaining the global lease index, handling cross-node lease conflicts, and dynamically adjusting lease permission delegation.

[0043] This layered architecture effectively leverages the strengths of each layer to create an efficient, low-latency, and scalable file access and lease management process, making it particularly suitable for large-scale, high-concurrency enterprise application environments. Local operations at the client layer minimize reliance on remote storage services, while RDMA optimization and hot replication mechanisms at the node layer improve data access performance and fault recovery capabilities. Finally, global state maintenance and coordination at the cluster layer ensure overall system data consistency and high availability.

[0044] The application scenarios of the solution in this application are not limited to the above examples; the method of this application can be used in any storage system.

[0045] The embodiments of this application provide a distributed lease management method. Figure 2 This is a flowchart of an optional distributed lease management method according to an embodiment of this application; as follows: Figure 2 As shown, the distributed lease management method includes:

[0046] Step S202: Upon receiving a file operation request, a lease request is sent to the node layer, wherein the lease request is used to indicate the target lease of the target file corresponding to the file operation request;

[0047] It's important to note that file operation requests are demands triggered by applications or users, such as reading, writing, or deleting files. In distributed storage systems, these requests must undergo lease acquisition and verification before any actual data operations are performed to ensure data consistency and security.

[0048] The node layer, or DFS-server, is a service located on each compute node. It manages lease requests from local processes, arbitrates lease conflicts, and communicates with the cluster layer (Monitor / MDS) to coordinate the global lease state. A lease request contains crucial information required for file operation requests, such as the target file path, operation type (read or write), requested lease type (read or write lease), and information about the requesting client. The lease request is a necessary step in acquiring a file lock, ensuring that multiple clients accessing the same file in a distributed environment can do so in an orderly and safe manner.

[0049] In an optional implementation, the LibFS-client acts as a bridge between the application and the distributed file system service (DFS-server) of the distributed storage system. When the application calls the POSIX file system API to perform file operations, the LibFS-client captures (intercepts) the call using hook technology, analyzes the nature of the operation and the target file, and then constructs an appropriate lease request, which includes the file path, operation type, and identification information of the requesting client.

[0050] Subsequently, the LibFS-client sends the lease request to the local DFS-server. This process can be achieved in various ways, including but not limited to local socket communication, shared memory, or inter-process communication (IPC) mechanisms. Once the DFS-server receives the lease request, it will begin local lease checks and status arbitration to determine whether the lease can be granted immediately or whether further communication with the cluster layer (Monitor / MDS) is needed to handle lease conflicts.

[0051] In specific implementations, LibFS-client can function as a user-space file system library, loaded via the LD_PRELOAD environment variable or statically linked to the application at compile time. By implementing a POSIX-compliant API, it seamlessly replaces the system's existing file system calls, meaning applications can utilize the lease management services provided by LibFS-client with little or no modification. During lease request construction and transmission, LibFS-client utilizes a pre-established communication channel with the DFS-server to ensure requests are quickly and accurately transmitted to the node layer for further lease processing and conflict management.

[0052] Step S204: After receiving the target lease sent by the node layer, the target file is accessed in the storage system according to the target lease. The node layer corresponds to at least one client layer. In the case of a conflict between the target lease and the reference node layer, the node layer accesses the cluster layer. When the node layer receives the authorization instruction sent by the cluster layer, the node layer sends the target lease.

[0053] It should be noted that the reference node layer refers to the node layer that holds a valid lease for the same file before the target lease request. A lease conflict between the target lease and the reference node layer means that two nodes are attempting to access and potentially modify the same file simultaneously, which requires further coordination to avoid data inconsistency.

[0054] The cluster layer consists of Monitor and MDS, which are responsible for global state consistency and coordinating lease conflicts across nodes. MDS can handle subtree leases and dynamically adjust lease management rights by maintaining a global lease index and metadata, while Monitor ensures the consistency of multi-node states through distributed consensus algorithms such as Paxos.

[0055] When a node layer receives a target lease, it verifies its validity and checks for conflicting reference node layer leases. If the target lease does not conflict with the reference node layer lease, the node layer allows the client layer to perform corresponding read or write operations on the target file based on the lease type. If a conflict exists, the node layer sends a request to the cluster layer, which arbitrates based on the global lease status. Possible actions include extending or shortening the lease validity period, notifying the original lease holder to release the lease, or reallocating the lease.

[0056] In an optional implementation, a file access operation initiated by the client layer first requires acquiring a target lease, a request that is sent to the local node layer. Upon receiving the lease request, the node layer first checks its local lease table to determine if the lease can be granted. If there are no conflicting leases locally, the target lease is granted and recorded in the local lease table; if a conflict exists, the node layer does not immediately reject the request but evaluates the conflict. If the conflict can be resolved internally by the node layer (e.g., the conflict occurs in the local cache), it is resolved through internal mechanisms; if the conflict involves other nodes—that is, the target lease conflicts with a lease on a reference node layer—the node layer sends a query request to the cluster layer, requesting the cluster layer to arbitrate the lease conflict.

[0057] The cluster layer, through the CRUSH algorithm which maintains the global lease status and replica data distribution, can check the current lease status of the target file to determine if cross-node lease conflicts actually exist. If no conflict exists, the cluster layer will authorize the node layer to grant the lease; if a conflict exists, the cluster layer will coordinate to resolve the conflict, which may include notifying the original lease holder to release the lease, or, if the CRUSH algorithm allows, pre-adjusting the replica placement to reduce subsequent lease conflicts. Once the conflict is resolved, the cluster layer will update the global lease status and authorize the target node layer to grant the lease.

[0058] Ultimately, after acquiring a lease, the client layer can operate on the target file based on the lease type (read or write) and permissions (such as lock or unlock). For write operations, the client layer first records the operation in the local NVM log, then synchronizes a hot copy via RDMA, and then asynchronously refreshes the data to the OSD to achieve low-latency write operations and high-availability data consistency.

[0059] Example 1:

[0060] Suppose client A attempts to write to file File1. Client A first requests a write lease from its local node layer, DFS-server1. DFS-server1 checks its local lease table and finds no conflicting leases, so it grants the write lease to client A and records it in its local lease table. After receiving the write lease, client A atomically records the write operation to its local NVM log, synchronizes the write operation to the NVM cache of the hot replica node DFS-server2 via RDMA, and then asynchronously flushes the data to the OSD storage device.

[0061] However, if another client B (via DFS-server3) is performing a write operation on File1 before DFS-server1 grants the write lease, and DFS-server1 detects a lease conflict between the target lease and the reference node layer (DFS-server3), DFS-server1 will not directly grant the lease. Instead, it will send a lease conflict query request to the cluster layer MDS. MDS checks the global lease state and finds that the write lease for File1 is indeed held by client B. Therefore, MDS notifies client B on DFS-server3 to release the lease (if after a grace period) and readjusts the replica location prediction for File1 to reduce subsequent lease conflicts. MDS then notifies DFS-server1 that the write lease can be granted. After receiving authorization from MDS, DFS-server1 grants the write lease to client A, who continues the write operation and updates the global state.

[0062] This application describes a system where, upon receiving a file operation request, the client layer sends a lease request to the node layer. This lease request indicates the target lease for the target file corresponding to the file operation request. After receiving the target lease from the node layer, the client layer accesses the target file in the storage system based on the target lease. Each node layer corresponds to at least one client layer. In the event of a conflict between the target lease and a reference node layer, the node layer accesses the cluster layer. By delegating lease request processing from the remote cluster layer to the local node layer, communication latency is significantly reduced, and the response speed of file operations is improved. When the client layer receives a file operation request, it intercepts the POSIX API and manages the lease request locally, avoiding the overhead of direct network round trips to the remote MDS or Monitor for each file operation. This results in more efficient overall system performance, especially for high-frequency read / write operations, significantly reducing latency. Once the client layer successfully obtains the target lease from the node layer, it can directly access the target file through the distributed storage system without further lock waiting or queries. This immediate access capability improves the efficiency of file operations, particularly for read operations, as acquiring read leases typically does not cause conflicts and can be completed quickly. For write operations, the existence of leases ensures exclusivity, thus avoiding data inconsistency and guaranteeing data integrity and security. Therefore, it can solve the problem of low resource utilization in distributed file systems of related technologies.

[0063] In an optional implementation, sending a lease request to the node layer includes: accessing a lease cache library, wherein a local lease cache is used to indicate at least one cached lease; if a target lease exists in the local lease cache, accessing a target file in the storage system based on the target lease; and if no target lease exists in the local lease cache, sending a lease request to the node layer.

[0064] It's important to note that the lease cache library is a local cache maintained by the client layer (LibFS-client) to store already acquired lease information. The lease cache library exists to reduce frequent requests to the node layer (DFS-server), improving the efficiency and response speed of file operations. The local lease cache, as part of the lease cache library, stores the lease records currently held or cached by each process, including information such as file path, lease type, expiration time, and version number. The local lease cache is a key data structure for implementing lease prefetching and access pattern prediction.

[0065] In an optional implementation, LibFS-client performs a rapid check of lease status by accessing a local lease cache, a crucial step before the entire file operation process. Specifically, when an application attempts to operate on a file, LibFS-client first queries the local lease cache to see if the lease for that file is already cached. If it exists and the lease is valid, LibFS-client can directly perform read or write operations on the file based on the lease type, without waiting for a response from the remote DFS-server, significantly improving the immediacy and response speed of the operation.

[0066] However, if the LibFS-client does not find the desired target lease in its local lease cache, or if the lease has expired, it will construct a lease request, containing information about the target file and the requester, and send it to the DFS-server via a local socket or shared memory mechanism. This step initiates the lease acquisition process, where the DFS-server will determine whether to grant the lease, as well as the lease type and validity period, based on its local state or the results of its interaction with the cluster-level MDS.

[0067] In an optional implementation, the local lease cache can be maintained using in-memory data structures such as hash tables or B-trees for fast lookups and updates. To ensure persistence in the event of a system crash or restart, critical data in the cache (e.g., lease information and version numbers) needs to be periodically persisted to NVM or disk. Upon receiving a file operation request, the LibFS-client first checks the local lease cache, a step that can be efficiently performed by querying data structures such as hash tables or B-trees. If the target lease exists and is valid, the LibFS-client directly performs the file operation; if the target lease does not exist or has expired, the LibFS-client constructs a lease request and sends it to the DFS-server. The lease request needs to include the identifier of the target file, the requested lease type (read or write), the client ID, and a possible callback address. This request can be sent to the DFS-server via a local socket communication mechanism, eliminating the need for network communication and reducing latency.

[0068] In a distributed storage system, an application requests a read operation on the file / data / log.txt. Upon receiving this request, the LibFS-client first accesses its local lease cache to check for a read lease for / data / log.txt. If a valid read lease is found in the cache, the LibFS-client will directly execute the read operation based on the lease's permissions, potentially reading data from the NVM cache, without sending an additional request to the DFS-server.

[0069] However, if the local lease cache does not contain a read lease record for ` / data / log.txt`, or if the recorded lease has expired, the LibFS-client will construct a read lease request, including the file path ` / data / log.txt` and the request type (read). The LibFS-client then sends the lease request to the local DFS-server via a local socket. Upon receiving the request, the DFS-server performs a local lease status check. If there are no conflicting leases locally, the DFS-server will grant the read lease and update the local lease table, while simultaneously updating the read lease information in the cache for future access. If a conflict exists, the DFS-server will send a query request to the MDS, which will then arbitrate the global lease status, resolve the conflict, and notify the DFS-server to further process the lease request.

[0070] This approach enables intelligent management of lease requests, avoids unnecessary remote communication, improves the efficiency of file operations and system performance, and ensures data consistency and security.

[0071] In an optional implementation, before accessing the target file in the storage system according to the target lease, the process includes: after receiving the target lease sent by the node layer, determining the operation instruction corresponding to the file operation request; if the operation instruction is a write operation, recording the write operation in the operation log corresponding to the node layer, wherein the operation log is used to indicate the operation type and data update information corresponding to at least one write operation.

[0072] It should be noted that operation instructions refer to commands issued from the perspective of the application or user, instructing the user to perform a certain type of access or modification to a file. These can specifically be commands from the POSIX API such as open, read, write, rename, and delete. A write operation refers to the write command included in a file operation request, i.e., the process of writing or updating data into a file.

[0073] The operation log resides at the node layer (DFS-server) and records detailed information about all write operations, including operation type, data update content, and operation timestamp. The operation log helps ensure operation persistence and transaction consistency; even in the event of node failure or system crash, the system state can be recovered by reviewing the operation log.

[0074] Data update information refers to data associated with write operations, including but not limited to the data block to be written, the offset and length of the data block, etc. They are an indispensable part of the operation log and are used to accurately describe the data update actions.

[0075] In an optional implementation, upon receiving a target lease, the LibFS-client first needs to determine the operation instruction corresponding to the file operation request. This typically involves parsing the operation type field in the lease request. If the operation instruction is determined to be a write operation, the LibFS-client will further execute the step of recording the write operation to the operation log. This step is achieved by using an atomic write operation (such as pmem_persist provided by libpmem) to persist information such as the type of the write operation, the target file location, the data content to be written, and the offset in NVM. Through this mechanism, the information of the write operation will not be lost due to any sudden power outages or system failures.

[0076] In an optional implementation, upon receiving a target lease, the LibFS-client parses the operation instructions in the lease request to identify whether it is a read, write, or other type of file operation. Using the `pmem_persist` function from libpmem or other similar libraries, it writes the specific information of the write operation (including operation type, filename, data update content, etc.) to the operation log. libpmem leverages NVM hardware features to ensure the persistence of data write operations, allowing recovery of the last state even in the event of an abnormal system shutdown. After recording the write operation locally, the LibFS-client synchronizes this write operation to the hot copy maintained by the DFS-server using RDMA (Remote Direct Memory Access) technology to maintain data consistency across multiple nodes. RDMA technology features low latency and high bandwidth, making it ideal for achieving efficient inter-node data transfer in a distributed environment.

[0077] Example 2:

[0078] Client C is performing a write operation on the file data.txt. Client C's application invokes the POSIX-compliant `write` command, which is intercepted by the LibFS-client library. LibFS-client first checks its local lease cache to confirm the validity of the write lease for data.txt. Then, it receives the target lease authorized from the cluster layer from the DFS-server. After determining that the operation is a write operation, LibFS-client records relevant information about this write operation (including the operation type, the specific data to be written, the path to the file data.txt, and the data offset) in its local operation log. This operation log is supported by NVM and uses the libpmem library to ensure the atomicity and persistence of the write operation.

[0079] Next, the LibFS-client uses RDMA technology to synchronize write operations to the hot replica nodes maintained by the DFS-server. This process leverages the direct memory access feature of RDMA, avoiding the additional encapsulation and decapsulation of the TCP / IP protocol, significantly reducing synchronization latency and improving the speed and efficiency of data synchronization.

[0080] Finally, after all the data from the write operation has been persisted on the local machine and the hot replica node, the LibFS-client asynchronously flushes the data to the OSD (Object Storage Device), completing the entire data write process. This series of actions ensures high performance and high reliability of the write operation, and also demonstrates the unique value of the multi-layer architecture in this invention in improving the performance of distributed storage systems. By using operation logs and the hot replica mechanism, even in the event of node failure, the system can quickly recover services and ensure data consistency and integrity.

[0081] In an optional implementation, after accessing the target file in the storage system according to the target lease, the method includes: updating the status of the write operation in the operation log to a first status, wherein the first status indicates that the write operation has been completed.

[0082] It's important to note that the first state refers to a marker in the operation log indicating that the write operation has been successfully completed and its result has been persisted. In distributed storage systems, updating this state is crucial for maintaining system consistency and integrity.

[0083] In an optional implementation, each line in the operation log corresponds to a specific file operation. For completed write operations, the DFS-server needs to update its status field from "in progress" to "completed," which typically involves adding or changing a status identifier in the log entry. To ensure the persistence of status updates, the DFS-server uses atomic write functions provided by libpmem or other NVM-compatible libraries, such as pmem_persist, to update the status field in the operation log. This ensures that even in the event of a system crash or power outage, the status information in the operation log is not lost, guaranteeing that the system can accurately understand the status of each file operation after a restart, thus enabling correct recovery operations. To prevent the log file from becoming too large, the DFS-server also periodically performs log compression operations, cleaning up completed entries, freeing up NVM space, and making room for new operation records. This helps maintain the efficiency and availability of the operation log and reduces the long-term occupation of NVM resources.

[0084] In a distributed storage system, client C is performing a write operation on the file log_file.txt. This operation is intercepted by LibFS-client, and the DFS-server and hot replica nodes complete the data writing, synchronization, and persistence. Once all these steps are complete, the DFS-server will look for the entry corresponding to the write operation on log_file.txt in the operation log.

[0085] Next, the DFS-server will use an atomic write operation, such as pmem_persist, to update the status of the entry from "in progress" to "completed," marking the formal end of the write operation. This update will ensure that even if the system encounters a sudden failure at this moment, such as a power outage, the status information in the operation log will be available upon the next system startup, indicating that the write operation to log_file.txt has been successfully completed.

[0086] In addition, the DFS-server periodically checks the entries in the operation log, compresses those marked as "completed," and removes them to reclaim NVM space. In this way, even under continuous high-concurrency write operations, the operation log remains at a reasonable size, avoiding performance degradation caused by log file bloat.

[0087] Through the above-described embodiments of this application, this management of operation log states, combined with the high-speed storage characteristics and hot replication mechanism of NVM, not only improves the efficiency and reliability of file operations in the distributed storage system, but also ensures system crash consistency. Even in abnormal situations, it can recover to the most recent persistent state, reducing the risk of data loss. Therefore, the technical solution of this invention not only optimizes the file lock lease management process, but also enhances the performance of the distributed storage system in the face of high concurrency and high complexity file access scenarios.

[0088] In an optional implementation, before accessing the lease cache library, the method includes: determining the access mode corresponding to the program, and determining at least one reference file based on the access mode; predicting at least one reference lease corresponding to the program based on the at least one reference file; and caching the at least one reference lease to the lease cache library.

[0089] It's important to note that access patterns refer to the regularity and frequency of a program's read or write operations on the file system. Access patterns can be sequential, random, write-intensive, read-intensive, etc., reflecting different file access needs of the application. Reference files are those files most likely to be accessed again, selected when predicting access patterns and lease demands. The selection of reference files is typically based on historical access records or file attribute analysis.

[0090] Reference leases are derived based on access pattern predictions and represent file leases that the application may need in the near future. Predicting reference leases helps to pre-lock files, reduce lease request latency, and improve file operation response speed.

[0091] In an optional implementation, the LibFS-client first continuously monitors application access to files, analyzes access patterns, and identifies files with high access frequency or importance as reference files. Then, it uses a pre-trained LSTM model to predict the leases (read or write leases) these reference files may require in subsequent operations. The prediction results indicate which leases should be requested in advance, as well as parameters such as the expected lease validity period.

[0092] Once the prediction is complete, the LibFS-client caches the predicted reference lease information in its local lease cache, awaiting the actual file operation. When a file operation request arrives, the LibFS-client first queries the lease cache to check if a valid lease exists for the target file. If no valid lease is found, it then sends a lease request to the DFS-server. Conversely, if a valid lease is found in the lease cache, the LibFS-client can directly use the lease without sending a remote request, significantly reducing waiting time and improving the immediacy of file operations and overall system performance.

[0093] In one example, a database application is running and needs to frequently access a database log file named log.db. After monitoring for a period of time, LibFS-client found that log.db is one of the most actively accessed files and therefore marked it as a reference file.

[0094] Next, LibFS-client uses an LSTM model to predict the type of lease the log.db file might need next. Since log.db typically requires continuous write operations, the prediction model is likely to infer that a write lease will be necessary. Based on the prediction results, LibFS-client requests a write lease from the DFS-server in advance and caches this write lease in its local lease cache.

[0095] When the application actually initiates a write operation on log.db, the LibFS-client immediately searches the lease cache and finds a valid write lease already cached. The LibFS-client then directly uses this write lease to record the write operation in the local NVM log and synchronizes the update to the hot replica node via RDMA. The entire process is fast and smooth, eliminating the need to send lease requests to the DFS-server, significantly reducing file operation waiting time and network communication latency.

[0096] Through the above-described embodiments of this application, applications can perform file operations with lower latency, especially for frequently accessed files such as database logs, where the performance improvement is particularly significant. The lease caching and intelligent prediction mechanisms of this invention effectively solve the bottleneck problem caused by lease management in traditional distributed file systems, greatly improving the system's responsiveness and overall performance.

[0097] In an optional implementation, after recording the write operation to the corresponding operation log of the node layer, the process includes: reading the operation log in the event of a restart; identifying at least one write operation in the second state of the operation log; sending a lease request to the node layer; and performing at least one write operation.

[0098] It should be noted that the second state in the operation log is used to indicate that the write operation has not yet been completed or acknowledged. Unlike the "first state" (write operation completed), write operations in the second state may not have been acknowledged or synchronized to the remote storage device due to abnormal interruption, and need to be identified and processed after the system restarts.

[0099] In an optional implementation, ensuring data integrity and consistency is crucial in the event of a system restart. The LibFS-client identifies write operations that were not acknowledged before the system crash by reading local NVM logs; these operations are marked as being in a second state. Once the LibFS-client identifies these incomplete write operations through the operation logs, it sends a new lease request to the DFS-server to obtain the leases required to perform these write operations.

[0100] Specifically, upon restarting, the LibFS-client uses libpmem or a similar library to read the operation log in NVM. During the log reading process, it searches for write operation records marked as "second state." For each such record, the LibFS-client constructs a new lease request, including the path to the target file, the required write operation type, and any necessary parameters, and then sends this request to the local DFS-server.

[0101] Upon receiving a lease request, the DFS-server performs local or remote arbitration of the lease to ensure that write operations do not conflict with read / write operations of other processes or nodes. Once the DFS-server grants the lease, the LibFS-client re-executes the write operation, updating the data in the NVM cache and synchronizing it to the hot replica node via RDMA technology, while asynchronously flushing it to the OSD. Afterward, the status of the corresponding operation in the operation log is updated to "first state" (write operation completed), ensuring data consistency and integrity are guaranteed even in the event of a system restart.

[0102] Example 3:

[0103] Suppose that during system operation, a client C is writing to the file critical_data.txt, and suddenly the system experiences an unexpected power outage due to a power fluctuation. After the system recovers and restarts, the LibFS-client first reads the operation log in the local NVM and identifies that the previous write operation to critical_data.txt was not confirmed due to the power outage, and the recorded status is "Second State".

[0104] The LibFS-client then sends a lease request to the local DFS-server, requesting a write lease for critical_data.txt to complete the interrupted operation. Upon receiving the request, the DFS-server checks its local lease table to confirm that no other conflicting leases exist, then grants the client C a write lease and updates the lease table.

[0105] With the write lease in place, the LibFS-client re-executes the write operation to critical_data.txt, writing the relevant data to the local NVM cache and synchronizing it to the hot replica node via RDMA to ensure data consistency across multiple replicas. Once the data is confirmed to be persistent on the hot replica node, the LibFS-client updates the status of the write operation in the local NVM log to "first state," i.e., completed, thus avoiding redundant writes or inconsistent states and ensuring data integrity and transaction durability.

[0106] Through the above-described implementation methods of this application, in the scenario of system power failure and restart, the consistency of write operations is guaranteed by the mechanism of NVM logs, lease requests, and re-execution of write operations, thus solving the data integrity problem that may occur in traditional distributed storage systems.

[0107] Figure 3 This is a flowchart of another optional distributed lease management method according to an embodiment of this application; such as Figure 3 As shown, this distributed lease management method is used at the node layer and includes:

[0108] Step S302: Receive a lease request sent by the client layer, wherein the lease request is used to indicate the target lease of the target file corresponding to the file operation request;

[0109] Step S304: Query the local lease table, and if the target lease is held by the reference node layer, send a query request to the cluster layer, wherein the query request is used to query the status of the target lease;

[0110] Step S306: Upon receiving the authorization instruction sent by the cluster layer, send the target lease to the client layer.

[0111] It should be noted that the local lease table is a data structure maintained by the DFS-server, recording information on all active leases on the current node, including lease holder, lease type, expiration time, etc. The local lease table is the primary basis for the node layer to perform lease status checks and conflict resolution.

[0112] The authorization instruction is a message sent from the cluster layer (Monitor / MDS) to the node layer, confirming the grant or denial status of a specific lease. If the cluster layer confirms that there are no conflicts with the leases of the target file, it will send an authorization instruction to the node layer, allowing the node layer to grant the target lease to the client layer.

[0113] In the implementation of the node layer (DFS-server), the above steps are achieved in the following way:

[0114] The DFS-server is configured with a listening mechanism that can receive lease requests from the client layer. Once a request arrives, the DFS-server will parse the file identifier and lease type in the request and prepare to check the lease status.

[0115] The DFS server maintains a local lease table to record the status of all active leases on the current node. Upon receiving a lease request, the DFS server queries the local lease table to determine if the target lease has already been granted to another client or process. If a lease conflict is detected, the DFS server does not immediately reject the request but instead sends a query request to the cluster layer for further guidance.

[0116] When a conflict exists between target leases in the local lease table, the DFS-server sends a query request over the network to the cluster layer (Monitor / MDS) to confirm the status of the target lease. This step ensures the consistency of the global lease status and the fairness of the decision-making through a multi-node distributed consensus protocol, such as Paxos or Raft.

[0117] Once the DFS-server receives the authorization instruction from the cluster layer, it parses the instruction content to determine whether the target lease can be granted to the requester. If the authorization instruction allows, the DFS-server updates its local lease table, records the holder information of the new lease, and sends the target lease to the client layer, allowing it to perform file operations.

[0118] Example 4:

[0119] Suppose client C is performing a write operation on the target file data.txt. Client C's LibFS-client cannot find a valid write lease in its local lease cache, so it sends a write lease request to the local DFS-server.

[0120] Upon receiving a lease request, the DFS-server first queries its local lease table and finds that the write lease for data.txt is already held by another client, D. In a conventional distributed file system, the DFS-server would directly reject client C's request in this situation. However, in this invention, the DFS-server chooses to send the lease request to the cluster layer, requesting MDS to arbitrate the global state.

[0121] Upon receiving the query request, MDS checks the global lease index and confirms that the write lease for data.txt is indeed held by client D, but client D's write operation has been completed and the lease has been released. Therefore, MDS sends an authorization command to the DFS-server, allowing the DFS-server to grant a write lease to client C.

[0122] After receiving the authorization command, the DFS-server updates its local lease table to record that the write lease for data.txt is held by client C, and then sends the write lease back to client C's LibFS-client. Upon receiving the write lease, the LibFS-client will be able to perform write operations on data.txt without having to go through the lease request waiting process again.

[0123] Through the steps described above, this application effectively solves the problem of lease conflict management, avoids unnecessary waiting and performance loss, and ensures strong consistency of file data in a distributed environment. This mechanism is particularly suitable for high-concurrency and large-scale data access scenarios, and can significantly improve the overall performance and efficiency of distributed storage systems.

[0124] In an optional implementation, after querying the local lease table, the following steps are included: if the operation instruction corresponding to the file operation request is a read operation, the target lease is sent to the client layer.

[0125] It should be noted that operation instructions are specific operation commands, such as read, write, and open, contained in file operation requests sent by the client layer in a distributed storage system. These instructions indicate the specific type of operation to be performed on the target file.

[0126] A read operation is a file operation command used to retrieve file contents. In a distributed storage environment, reading operations often involve checking locking mechanisms to ensure data consistency.

[0127] A target lease is a lease that the client layer needs to acquire in order to access a file during read operations. A target lease allows the client layer to read file data without write conflicts.

[0128] Upon receiving a file operation request at the client or node layer, the system first queries the local lease table to check if a lease already exists on the target file, and the type and status of the lease. This check is crucial for ensuring data consistency in a distributed environment. If the file operation request is determined to be a read operation, and the query of the local lease table finds either no lease on the target file or only a read lease, then the node layer will send the target read lease to the client layer. This process allows the client layer to directly read the file without write conflicts, improving the response speed and efficiency of read operations.

[0129] In an optional implementation, during the normal operation of the distributed storage system, when the client layer receives a read operation request from the application, the LibFS-client first queries its local lease cache to check if it already holds a read lease for the target file. If a valid read lease exists in the local cache, the LibFS-client can directly execute the read operation without further communication with the node or cluster layer, thereby greatly reducing network communication overhead and improving the response speed of read operations.

[0130] However, if the LibFS-client finds that the read lease for the target file does not exist or has expired after querying the local lease cache, it will send a read lease request to the DFS-server. Upon receiving the read lease request, the DFS-server checks the node-level lease table to confirm that no write lease exists for the target file, or that the current node is the holder of the write lease. If the conditions are met, the DFS-server will grant the read lease and send the target read lease back to the client layer through the communication channel with the LibFS-client. Once the read lease is granted, the LibFS-client can perform read operations on the file. The data read will first be retrieved from the local NVM cache; if the local cache does not contain the required data, it will be loaded asynchronously from a hot copy or OSD.

[0131] The above-described embodiments of this application effectively improve the performance of read operations in distributed storage systems, reduce reliance on remote storage devices, and ensure data consistency and security through a lease mechanism. In high-concurrency read scenarios, this mechanism can significantly reduce network communication latency and improve the overall read throughput of the system.

[0132] In an optional implementation, after querying the local lease table, the process includes: determining the holding status of the target lease if the operation instruction corresponding to the file operation request is a read operation; sending the target lease to the client layer if the holding status indicates that the target lease is not held; and placing the file operation request into a waiting queue if the holding status indicates that the target lease is held by the reference client layer.

[0133] It's important to note that, within the distributed file lock mechanism, the holding status refers to the holding status of the target lease for a given file operation request—that is, whether the target lease is currently held by any client or process, and the specific information of the holder. The waiting queue is a data structure maintained by the DFS-server to store file operation requests that cannot be immediately authorized while the target lease is currently held by another client. These requests will be reprocessed according to a specific strategy and order after the target lease is released.

[0134] In an optional implementation, upon receiving a read operation request, the DFS-server immediately accesses its local lease table to check if a read lease for the target file is available. If the read lease is not held by any client, the DFS-server grants the read lease directly, and the LibFS-client can then begin reading the file.

[0135] However, if a read lease is already held by another client, even if the operation is a read operation, the DFS-server will place the request in a waiting queue instead of immediately granting the read lease. This is to prevent read-write or read-read conflicts in scenarios with concurrent access from multiple clients, especially when the target file is under write operation, as the result of the read operation may become inconsistent due to the intermediate state of the write operation. In this case, the DFS-server will record the callback information from the LibFS-client and automatically retrieve the request from the waiting queue after the target lease is released or expires, re-grant the lease, and notify the LibFS-client.

[0136] Suppose client A is about to read a file named report.doc. Client A's LibFS-client library first checks its local lease cache and finds no valid read lease. Therefore, LibFS-client sends a read lease request to the local DFS-server. Upon receiving the request, the DFS-server queries its local lease table and discovers that the read lease for report.doc is actually held by client B, and client B is currently writing to that file.

[0137] In this scenario, the DFS server will neither directly reject client A's read request nor immediately grant a read lease. Instead, the DFS server will place client A's read operation request in a waiting queue and record client A's callback information. The DFS server will then monitor client B's write operation status until the write operation is completed or client B actively releases the read lease.

[0138] Once client B completes its write operation and releases its read lease, the DFS-server retrieves client A's read request from the waiting queue, checks the current lease status to ensure there are no new conflicts, grants client A a read lease, and notifies the LibFS-client via a callback mechanism. After receiving the read lease, client A can begin reading the contents of the report.doc file without worrying about data version inconsistencies or conflicts.

[0139] The above implementation methods not only ensure strong consistency in file operations, but also reduce unnecessary client waiting time, thereby improving the overall system response speed and user experience.

[0140] In an optional implementation, after sending the target lease to the client layer, the process includes: updating the target lease and the client layer to the local lease table; sending the local lease table to at least one reference node layer; and updating the cache file of the node layer based on a file operation request.

[0141] It should be noted that the reference node layer in a distributed storage system includes not only the node directly involved in the lease request (the current node), but also nodes with hot replica relationships to the current node. The hot replica mechanism is used to maintain data consistency and redundancy across multiple nodes, and typically involves leases and data synchronization at least one reference node layer.

[0142] The cache file is a copy of file data maintained by the DFS-server in local NVM (non-volatile memory). It is used to quickly respond to client read operation requests, reduce reliance on remote storage devices, and improve overall system performance. Updates to the cache file are based on file operation requests, ensuring consistency between the local cache and remote storage data.

[0143] In an optional implementation, after granting a target lease, the DFS-server immediately updates its local lease table, updating the holder of the target lease to the corresponding client layer, and recording detailed lease information, including validity period and version number. Subsequently, the DFS-server broadcasts the updated local lease table information to all reference node layers via RDMA technology, ensuring that the hot replica node layers are also synchronized with the latest lease status.

[0144] For file operation requests, the DFS-server updates the local cache file based on the operation type (read or write). If it is a write operation, the DFS-server records the update operation in the local NVM and synchronizes the update to the hot replica node via RDMA to ensure data consistency across all hot replicas. At the same time, the DFS-server asynchronously flushes the data to the OSD to complete the persistent storage of the data.

[0145] For read operations, the DFS-server will first read data from the local cache file. If the required data is not in the local cache or the data has expired, the DFS-server will obtain the latest file data from the hot replica node or OSD and update the local cache file in preparation for subsequent read requests.

[0146] Client C requests a write lease for the file data.txt to perform a write operation. After confirming there are no conflicts, the DFS-server sends the target write lease to client C.

[0147] After granting the write lease, the DFS-server updates its local lease table, recording that the write lease for data.txt is held by client C, and sets the lease's expiration date. Subsequently, the DFS-server synchronizes the updated information in its local lease table to the hot replica node layer associated with data.txt, such as node D, via RDMA.

[0148] When client C performs a write operation, the DFS-server records the details of the write operation, including data blocks and offsets, and uses RDMA to update and synchronize this data to the hot replica node D, ensuring file data consistency across all hot replica nodes. Simultaneously, the DFS-server asynchronously flushes the data to the OSD for persistent storage.

[0149] When another client E requests to read data.txt, the DFS-server will first read the data from the local cache file. If the local cache does not contain the latest data or the data has expired, the DFS-server will retrieve the latest data.txt file data from the hot replica node D or OSD, update the local cache, and then send the read result to client E.

[0150] Through the above implementation methods, the distributed file lock lease management method not only effectively solves the problems of lease conflict and data consistency, but also greatly improves the response speed of file operations and the overall system performance. It is an efficient solution for high-concurrency access and large-scale distributed storage scenarios.

[0151] In an optional implementation, after sending the local lease table to at least one reference node layer, the method includes: determining a candidate node layer in the at least one reference node layer if the node layer stops operating; determining at least one client corresponding to the node layer based on the local lease table, and establishing a connection between the candidate node layer and the at least one client; and sending a lease to the at least one client based on the lease request of the at least one client and the local lease table.

[0152] It should be noted that the alternative node layer is the node layer selected from the reference node layer that can take over its functions and tasks when the current node layer (DFS-server) stops running.

[0153] In an optional implementation, during the fault recovery process of a distributed storage system, when the DFS-server stops running for any reason, the system needs to quickly determine a backup node layer to take over the lease management responsibilities of the original node. The specific steps for implementing this mechanism are as follows:

[0154] After a DFS-server stops running, the system selects a healthy and low-loaded backup node layer from the reference node layer that it communicates with. This selection process is typically based on monitoring information and algorithms at the cluster layer (Monitor / MDS), such as election mechanisms or load balancing strategies. The backup node layer obtains the local lease table of the stopped DFS-server and analyzes the lease status and holder information. This step is crucial for determining which clients need to be notified and have their lease connections re-established. Based on the information in its local lease table, the backup node layer proactively establishes communication connections with the affected clients, informing them of the change in the original DFS-server's status and offering itself as the new lease service provider. The backup node layer handles the legacy lease requests from the original DFS-server, sending new leases to the clients based on the status of its local lease table and the clients' lease needs, allowing them to continue file read and write operations while ensuring the consistency and security of the distributed storage system.

[0155] Example 5:

[0156] In a distributed storage system, suppose the DFS server of node A suddenly stops running, and nodes B and C are reference nodes communicating with node A. Through coordination at the cluster layer, the system determines node B as the backup node to take over the lease management responsibilities of node A.

[0157] Node B first retrieves Node A's local lease table and analyzes it. It discovers that two clients, C1 and C2, are holding leases on Node A and are performing read and write operations on the files data1.txt and data2.txt, respectively. Subsequently, Node B establishes communication connections with clients C1 and C2, informing them of Node A's failure and offering itself as a new lease service node.

[0158] Node B processes legacy lease requests from Node A based on the status of its local lease table. For example, client C1's read lease request for data1.txt and client C2's write lease request for data2.txt. After confirming the lease status, Node B sends a read lease for data1.txt to client C1, allowing it to continue reading; simultaneously, it sends a write lease for data2.txt to client C2, ensuring its write operations can proceed safely.

[0159] Through the above implementation methods, even in the event of node-level failures, file access services can be quickly restored in a distributed storage system, ensuring the continuity of client operations and data consistency, thus greatly improving system high availability and user experience. This process achieves seamless fault recovery through automated fault detection, node election, lease state transition, and client notification mechanisms, reducing system downtime and data recovery delays.

[0160] In an optional implementation, after placing the file operation request into the waiting queue, the process includes: determining the target lease corresponding to the file operation request if the target lease held by the reference client is released; querying the local lease table and, if the holding status indicates that the target lease is not held, sending the target lease to the client layer.

[0161] It's important to note that the waiting queue is a data structure in the DFS-server used to store file operation requests that cannot be processed temporarily due to lease conflicts. When the target lease is held by another client, subsequent file operation requests will be placed in the waiting queue until the target lease is released, at which point the DFS-server will have a chance to process and respond to them.

[0162] In an optional implementation, during the DFS-server's lease management process, once a conflict is detected due to the target lease being held by another client, it places the current file operation request in a waiting queue instead of responding or rejecting it immediately. When the target lease held by the referencing client is released, the DFS-server retrieves all file operation requests related to the target lease from the waiting queue and then checks the status of the local lease table to confirm that the target lease is no longer held by any client.

[0163] If the target lease is not held, the DFS-server will grant the appropriate lease based on the type of file operation request (read or write) and send the target lease to the corresponding client through the communication channel with the client layer, allowing it to perform file operations. Simultaneously, the DFS-server will update its local lease table to record the new lease holder and lease validity period, ensuring that subsequent lease requests and conflict resolution are based on the latest lease status.

[0164] When handling the waiting queue, the DFS-server may also need to re-evaluate the priority of file operation requests and adjust the processing order of requests in the waiting queue based on information such as the client's request type and lease requirements, in order to optimize the overall system response time and resource utilization.

[0165] In one example, in a distributed storage system, client A and client B both attempt to write to the same file, shared_file.txt. Client A first requests and obtains a write lease for shared_file.txt and begins writing data. Subsequently, client B attempts to request a write lease for shared_file.txt, but the DFS server detects a lease conflict and places client B's request in a waiting queue, awaiting the release of the target lease.

[0166] After client A completes its write operation and releases its write lease, the DFS server no longer detects lease conflicts on shared_file.txt. Next, the DFS server retrieves client B's file operation request from the waiting queue, re-checks the status of its local lease table, and confirms that the write lease for shared_file.txt is currently not held by any client.

[0167] After confirming the availability of the target lease, the DFS server will send the target write lease to client B, allowing client B to perform a write operation on shared_file.txt. Simultaneously, it will update the information in the local lease table, recording client B as the current holder of the write lease. In this way, client B can continue its write operations while ensuring overall system data consistency and avoiding unnecessary waiting and performance degradation.

[0168] The above implementation method solves the lease conflict problem when multiple clients concurrently access shared files, optimizes the system's response time and resource utilization, and has a significant effect on improving the performance of distributed storage systems in high-concurrency scenarios.

[0169] Figure 4 This is a flowchart of another optional distributed lease management method according to an embodiment of this application; such as Figure 4 As shown, the distributed lease management method includes:

[0170] Step S402: Receive a query request sent by the node layer, wherein the query request is used to query the status of the target lease, and the target lease is the lease of the target file corresponding to the file operation request processed by the client layer;

[0171] Step S404: Determine the global lease index based on the query request, wherein the global lease index is used to indicate the local lease tables corresponding to the node layer and at least one reference node layer;

[0172] Step S406: If the reference node layer releases the target lease, an authorization instruction is sent to the node layer, wherein the authorization instruction is used to indicate that if the node layer receives the authorization instruction sent by the cluster layer, the node layer is allowed to send the target lease to the client layer.

[0173] It should be noted that the global lease index is a data structure maintained by the cluster layer (Monitor / MDS). It records the lease status and related information of all target files in the distributed storage system, including lease holder, lease type, lease validity period, etc. The global lease index is the key to achieving global consistency of lease status.

[0174] A query request is a request sent from the node layer (DFS-server) to the cluster layer (Monitor / MDS) to inquire about the current status of a specific target lease. Query requests typically contain information such as the identifier of the target file and the ID of the requesting node.

[0175] The authorization command is a command sent from the cluster layer (Monitor / MDS) to the node layer (DFS-server), instructing the node layer to grant the target lease to the client layer based on the state of the global lease index. The authorization command is the basis for the node layer to process lease requests, ensuring the global consistency of lease state in the distributed system.

[0176] The cluster layer (Monitor / MDS) receives query requests from the node layer (DFS-server). These requests contain the identifier of the target file and are used to query the current holding status of the target lease. Receiving the query request is the first step in lease status arbitration, ensuring that the cluster layer can make decisions based on a global view.

[0177] Upon receiving a query request, the cluster layer accesses the global lease index to check the lease status of the target file, including the current lease holder, lease type, and whether it has expired. This step ensures the correctness of the decision and avoids inconsistencies and conflicts that may result from local views at the node level.

[0178] If the cluster layer finds in the global lease index that the target lease has already been released, or that the current lease holder of the target file is the node layer that initiated the query, then it will send an authorization command to the node layer, instructing the node layer to grant the target lease to the client layer. Sending the authorization command is the final step in lease status arbitration; it enables rapid transmission and updating of lease status from the cluster level to the node level, ensuring the efficiency and consistency of the lease management process in the distributed system.

[0179] In an optional implementation, during distributed file lock lease management, the cluster layer (Monitor / MDS) acts as the arbitrator of the global lease status. When a node layer (DFS-server) discovers a conflict between the status in its local lease table and the actual requirements while processing a lease request—for example, the lease for the target file is held by another node layer, or the lease status is unknown—it sends a query request to the cluster layer for further decision-making and status confirmation.

[0180] Upon receiving a query request, the cluster layer accesses its global lease index to check the current lease status of the target file. If the target lease has been released by the holder, or if the cluster layer determines that the node initiating the query should be the lease holder, it generates an authorization instruction and sends it to the node layer. The authorizing node layer can then grant the target lease to the client layer.

[0181] In practice, the cluster layer uses distributed consensus algorithms such as Paxos or Raft to ensure the consistency of the global lease index updates and state. Upon receiving a query request from the node layer, MDS checks the global lease index to identify the lease holder and state of the target file. If the target lease is confirmed to be released, MDS immediately generates an authorization instruction and sends it to the requesting DFS-server via a high-speed network communication mechanism, allowing it to grant the lease to the client layer. Simultaneously, it updates the state information in the global lease index and the local lease table.

[0182] Example 6:

[0183] In a distributed storage system, client C requests a read lease for the file critical.log, but the DFS-server finds in its local lease table that the read lease for this file is already held by another node layer (DFS-server). At this point, the DFS-server does not immediately reject client C's request, but instead sends a query request to the cluster layer (MDS) to inquire about the global lease status of the critical.log file.

[0184] Upon receiving a query request, MDS accesses the global lease index and checks the current lease holder of critical.log. Suppose that at the time of the MDS check, the read lease holder of critical.log (another DFS-server) has just completed its operation on the file and released the lease. MDS confirms that the read lease status of critical.log is unheld, and it immediately generates an authorization instruction and sends it to the DFS-server that initiated the query.

[0185] After receiving the authorization instruction from MDS, the DFS-server updates its local lease table, marks the read lease status of the critical.log file as its own, and sends the target read lease to client C. Upon receiving the read lease, client C can directly perform a read operation on the critical.log file without waiting for the lease arbitration process.

[0186] The above implementation method solves the problem of rapid arbitration of lease conflicts, ensures global consistency of lease status, and improves the efficiency of lease requests and processing. Especially in high-concurrency access and cross-node conflict scenarios, this mechanism can significantly reduce waiting time and improve system performance and user experience.

[0187] In an optional implementation, when the reference node layer releases the target lease, an authorization instruction is sent to the node layer, including: sending a coordination instruction to the reference node layer holding the target lease; and sending an authorization instruction to the node layer when the reference node layer releases the target lease or the holding time of the target lease exceeds a preset time.

[0188] It should be noted that the coordination instruction is a specific authorization instruction used to coordinate between the lease holder and the client requesting the lease, especially when resolving lease conflicts. It is sent by the cluster layer to the reference node layer holding the target lease to guide the reference node layer to release the lease or perform other lease management operations.

[0189] The preset time is the validity period of the lease or the upper limit of the request waiting time in the waiting queue, and is dynamically set by the system administrator or an automatic adjustment algorithm. When the lease holding time exceeds the preset time, the system needs to take measures, such as forcibly canceling the lease, to avoid long file access wait times.

[0190] When the reference node layer (the DFS-server holding the target lease) releases the target lease, it means that the lease is no longer held and can be reassigned to other clients requesting the lease. At this point, the cluster layer (usually the MDS) sends an authorization instruction to the node layer, instructing it to grant the target lease to the next client in the request queue.

[0191] In some cases, the target lease may be held for longer than a preset time, or multiple clients may be waiting to acquire the same lease. To resolve lease conflicts more efficiently, the cluster layer will send a coordination instruction to the reference node layer that currently holds the target lease, instructing it to release the lease or adjust the lease holding status so that it can respond to requests from other clients more quickly.

[0192] If the target lease is held for more than the preset time, or after the lease is released, the cluster layer will send an authorization instruction to the node layer based on the global lease status and system load, to decide whether to immediately grant the target lease to the next client in the waiting queue, or to take other actions, such as recalculating the lease validity period or optimizing the lease allocation strategy.

[0193] In optional implementations, in distributed storage systems, lease management involves not only lease allocation and revocation but also handling dynamic authorization after lease release and coordination of lease conflicts. Specifically, the cluster layer (Monitor / MDS) acts as a global state manager and lease arbitrator. When a reference node (the DFS-server holding the target lease) releases the target lease, the cluster layer immediately sends an authorization instruction, instructing the node layer to reallocate the lease or perform other necessary operations, such as updating the lease expiration. If the lease is held for too long, the cluster layer sends a coordination instruction to the reference node layer, requesting it to release the lease to resolve potential lease conflicts. If the reference node layer responds and releases the lease, the cluster layer sends an authorization instruction to the node layer, authorizing it to handle the next lease request in the waiting queue.

[0194] In terms of implementation, the cluster layer maintains a global lease index through the Paxos protocol or a similar consensus algorithm, and monitors the holding status and duration of all leases in real time. When it detects that a lease has been released or its holding time has exceeded a preset time, it calculates the best course of action, such as reallocating the lease or adjusting the lease parameters, and sends instructions to the reference node layer and the node layer through a high-speed network to execute authorization or coordination operations.

[0195] In one example, suppose in a distributed storage system, client X requests a write lease for the file critical_data.csv, but the write lease for this file is currently held by the reference node layer Y (DFS-server). After waiting for a preset time (e.g., 3 seconds), client X's write request is still not processed because the reference node layer Y's lease holding time has exceeded the preset time.

[0196] At this point, the cluster-level MDS detected that the lease holding time of critical_data.csv was too long, triggering the lease conflict coordination mechanism. MDS sent a coordination command to the reference node layer Y, requesting it to release the write lease of critical_data.csv. Reference node layer Y responded to the coordination command and released the lease.

[0197] Next, MDS sends an authorization instruction to node layer Z (DFS-server), instructing it to grant a write lease for critical_data.csv to client X. After receiving the authorization instruction, node layer Z checks its local lease table to confirm that the target lease is not already held, and then sends a write lease to client X, allowing it to perform write operations on critical_data.csv.

[0198] Through the above implementation methods, even in high-concurrency scenarios, the distributed file lock lease management method of the present invention can respond to file operation requests in a timely manner, avoid long waiting times, and ensure a balance between system performance and data consistency through a dynamic mechanism of coordination instructions and authorization instructions.

[0199] In an optional implementation, before receiving a query request from the receiving node layer, the process includes: determining the type of the target file in the file operation request if the client layer is accessing the target file; and locking multiple files in the directory if the target file is a directory.

[0200] It's important to note that the type of the target file is important; files in a distributed file system can be of various types, including regular files and directories. Determining the type of the target file is crucial for choosing the correct lease management strategy.

[0201] In a file system, a directory is a container used to organize and store other files or directories. In lease management, access operations to a directory may require locking all its child files to ensure consistency and security.

[0202] This means that before the MDS (cluster layer) receives a query request from the DFS-server, the MDS needs to perform some preprocessing on the file operation request to ensure that subsequent lease processing and conflict coordination can be carried out quickly and efficiently. When the MDS detects a file operation request from the client layer, it first determines whether the target of the request is a directory or a regular file. This step is crucial in deciding how to handle the lease request, because access to a directory may require more complex lease management strategies to ensure the integrity of the directory structure. If the target file is determined to be a directory, the MDS will further lock the leases of all subfiles under that directory; this is called the subtree lease mechanism. This is done to ensure that during directory-level operations (such as renaming or deleting multiple files in a directory), other clients cannot perform concurrent read and write operations on these subfiles, thereby ensuring the consistency and security of the operations.

[0203] In an optional implementation, in distributed file lock lease management, when MDS receives a query request from the node layer, it first checks the target file identifier in the request and determines the file type. For directories, MDS performs subtree lease locking operations, pre-calculates the storage locations of all subfiles under the directory using the CRUSH algorithm, and then sends lock requests to the DFS-server of these subfiles to ensure that no concurrent access conflicts occur when performing directory-level operations.

[0204] Meanwhile, MDS maintains a global lease index, recording the current lease status of all files, including information such as lease holder, lease type, and expiration time. This allows MDS to quickly respond to node-level query requests, providing the latest lease status of the target file so that the node layer can decide whether to grant a lease or how to handle lease conflicts.

[0205] For directory locking, MDS automatically sends lock requests to the DFS-servers of all files within the directory, based on the subtree lease management policy. These requests require the DFS-servers to lock the files within a specified time window, disallowing other lease requests. Once the directory-level operation is complete, MDS releases the lock and notifies all DFS-servers to update the lease status.

[0206] In one example, suppose client C requests to rename the directory / projects, which requires locking all subfiles and subdirectories under / projects. Client C's application first calls the renameAPI, LibFS-client captures this request, and sends a directory-level write lease request to the local DFS-server.

[0207] The DFS server detected a conflict in the local lease table because another client, D, was writing to a subfile under / projects. Therefore, the DFS server sent a query request to the cluster layer (MDS).

[0208] After receiving a query request from a DFS server, MDS first determines that / projects is a directory. Then, MDS quickly calculates the DFS server locations for all subfiles under / projects using the CRUSH algorithm and sends locking commands to these DFS servers, instructing them to lock all subfiles under the directory and disallow any new read / write lease requests.

[0209] After all subfiles are locked, MDS sends an authorization command to the DFS-server, allowing the DFS-server to grant a directory-level write lease to client C. Upon receiving the write lease, LibFS-client performs a renaming operation on / projects and updates the local NVM log status to "completed".

[0210] Once the renaming operation is complete, the DFS-server will notify the MDS to release the subtree leases. Based on the release notification from the DFS-server, the MDS will unlock all subfiles and update the global lease index to ensure that subsequent file operation requests can proceed smoothly.

[0211] The above implementation method effectively solves the lease conflict problem in directory-level operations in a distributed environment. By locking subtree leases and global state arbitration of MDS, the system ensures data consistency and linearity of operation order when performing directory-level operations, which is especially suitable for large-scale, high-concurrency distributed storage system scenarios.

[0212] In an optional implementation, before receiving a query request sent by a node layer, the method includes: determining a conflict probability threshold, wherein the conflict probability threshold is used to indicate the probability that a node layer and at least one reference node layer request a lease for the same file; and determining the lease validity period of a target file based on the conflict probability threshold, the number of client layers included in the node layer and at least one reference node layer, and the operation frequency of the target file, wherein the lease validity period is used to indicate the valid time for a client layer to hold a target lease.

[0213] It should be noted that the conflict probability threshold is a metric set by the system administrator to measure the likelihood of lease conflicts occurring for target files in scenarios with concurrent access from multiple clients. Conflict probabilities below this threshold are considered acceptable, while those above the threshold may require measures to optimize lease validity periods to reduce the conflict rate.

[0214] The operation frequency of a target file refers to the number of times the target file is accessed (read or written) per unit of time, such as per second. Operation frequency is one of the important bases for dynamically adjusting the lease validity period. Files with high-frequency operations may require shorter lease validity periods to reduce conflicts, while files with low-frequency operations can have longer lease validity periods.

[0215] In an optional implementation, the cluster layer needs to determine a conflict probability threshold before calculating the lease validity period. This threshold reflects the highest level of lease conflict that can be tolerated when multiple clients concurrently access the distributed storage system, and is usually based on the specific requirements and strategy settings of the system, such as performance priority, consistency priority, etc.

[0216] After determining the conflict probability threshold, the cluster layer calculates the optimal lease validity period for the target file based on this threshold, the number of clients contained in the nodes and reference nodes in the current system, and the average operation frequency of the target file. This validity period calculation aims to balance system performance and data consistency, ensuring that the lease meets access requirements while minimizing the occurrence of conflicts.

[0217] Based on a Poisson process-based lease conflict probability model, the cluster layer calculates the lease validity period T of the target file. Once T is calculated, the cluster layer passes this parameter to the node layer via an authorization command, instructing it to follow the lease validity period standard when processing lease requests for the target file. After receiving the authorization command, the DFS-server updates its local lease management policy to ensure that all lease requests are processed according to the latest validity period standard.

[0218] In one example, in a distributed storage system, a log file named logs.txt is a shared resource among multiple containerized applications and is frequently accessed and written to daily. The cluster layer discovers that logs.txt is accessed an average of 100 times per second, while there are currently 50 clients (application instances) in the system that may access this file.

[0219] To reduce lease conflicts, the cluster layer sets a low conflict probability threshold (i.e., a 1% conflict probability). Using the lease validity period calculation formula described above, the cluster layer calculates the optimal lease validity period T for logs.txt to be 1 second.

[0220] Subsequently, the cluster layer sends this calculation result to all DFS-server nodes in logs.txt via an authorization command, including the node layer itself and at least one reference node layer. Upon receiving the authorization command, the DFS-server updates the lease validity period parameter in its local lease table to ensure that the granted lease validity period is 1 second when processing lease requests from logs.txt.

[0221] The above implementation method allows for dynamic adjustment of lease validity periods to cope with access pressure and system load of different file types, significantly reducing the risk of lease conflicts and improving overall system performance and user experience. For frequently accessed files like logs.txt, shorter lease validity periods reduce potential conflicts, while for less frequently accessed files, longer lease validity periods can be maintained to reduce unnecessary lease requests and processing overhead, achieving the optimal balance between performance and consistency.

[0222] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0223] Embodiments of this application also provide a distributed lease management device. Figure 5 This is a structural block diagram of an optional distributed lease management device according to an embodiment of this application, such as... Figure 5 As shown, for the client layer, it includes:

[0224] The request sending module 502 is used to send a lease request to the node layer when a file operation request is received, wherein the lease request is used to indicate the target lease of the target file corresponding to the file operation request;

[0225] Access module 504 is used to access the target file in the storage system according to the target lease after receiving the target lease sent by the node layer. The node layer corresponds to at least one client layer. In the case of a conflict between the target lease and the reference node layer, the node layer accesses the cluster layer. When the node layer receives the authorization instruction sent by the cluster layer, the node layer sends the target lease.

[0226] Optionally, the request sending module 502 described above is further configured to: access a lease cache library, wherein the local lease cache is used to indicate at least one cached lease; if a target lease exists in the local lease cache, access the target file in the storage system according to the target lease; if no target lease exists in the local lease cache, send a lease request to the node layer.

[0227] Optionally, the access module 504 is further configured to: after receiving the target lease sent by the node layer, determine the operation instruction corresponding to the file operation request; if the operation instruction is a write operation, record the write operation in the operation log corresponding to the node layer, wherein the operation log is used to indicate the operation type and data update information corresponding to at least one write operation.

[0228] Optionally, the access module 504 is further configured to: update the status of the write operation in the operation log to a first status, wherein the first status is used to indicate that the write operation has been completed.

[0229] Optionally, the request sending module 502 is further configured to: determine the access mode corresponding to the program, and determine at least one reference file according to the access mode; predict at least one reference lease corresponding to the program according to the at least one reference file; and cache at least one reference lease to the lease cache library.

[0230] Optionally, the access module 504 described above is further configured to: read the operation log in the event of a restart; identify at least one write operation in the second state of the operation log; send a lease request to the node layer and perform at least one write operation.

[0231] Embodiments of this application also provide another distributed lease management device. Figure 6 This is a structural block diagram of an optional distributed lease management device according to an embodiment of this application, such as... Figure 6 As shown, for the node layer, it includes:

[0232] The request receiving module 602 is used to receive a lease request sent by the client layer, wherein the lease request is used to indicate the target lease of the target file corresponding to the file operation request;

[0233] The lease table query module 604 is used to query the local lease table and, if the target lease is held by the reference node layer, send a query request to the cluster layer. The query request is used to query the status of the target lease.

[0234] The lease sending module 606 is used to send the target lease to the client layer upon receiving an authorization instruction from the cluster layer.

[0235] Optionally, the lease table query module 604 described above is also used to: send the target lease to the client layer when the operation instruction corresponding to the file operation request is a read operation.

[0236] Optionally, the lease table query module 604 is further configured to: determine the holding status of the target lease when the operation instruction corresponding to the file operation request is a read operation; send the target lease to the client layer when the holding status indicates that the target lease is not held; and put the file operation request into the waiting queue when the holding status indicates that the target lease is held by the reference client layer.

[0237] Optionally, the lease sending module 606 is further configured to: update the target lease and the client layer to the local lease table; send the local lease table to at least one reference node layer; and update the cache file of the node layer based on the file operation request.

[0238] Optionally, the lease sending module 606 is further configured to: determine a candidate node layer in at least one reference node layer when the node layer stops running; determine at least one client corresponding to the node layer according to the local lease table, and establish a connection between the candidate node layer and at least one client; and send a lease to at least one client based on the lease request of at least one client and the local lease table.

[0239] Optionally, the lease table query module 604 is further configured to: determine the target lease corresponding to the file operation request when the target lease held by the reference client is released; query the local lease table and send the target lease to the client layer when the holding status indicates that the target lease is not held.

[0240] Embodiments of this application also provide yet another distributed lease management device. Figure 7 This is a structural block diagram of an optional distributed lease management device according to an embodiment of this application, such as... Figure 7 As shown, the cluster layer includes:

[0241] The query receiving module 702 is used to receive query requests sent by the node layer. The query request is used to query the status of the target lease, and the target lease is the lease of the target file corresponding to the file operation request processed by the client layer.

[0242] The index determination module 704 is used to determine the global lease index based on the query request, wherein the global lease index is used to indicate the local lease table corresponding to the node layer and at least one reference node layer;

[0243] The authorization module 706 is used to send an authorization instruction to the node layer when the reference node layer releases the target lease. The authorization instruction is used to indicate that the node layer is allowed to send the target lease to the client layer when it receives the authorization instruction sent by the cluster layer.

[0244] Optionally, the authorization module 706 is further configured to: send a coordination instruction to the reference node layer holding the target lease; and send an authorization instruction to the node layer when the reference node layer releases the target lease or the holding time of the target lease exceeds a preset time.

[0245] Optionally, the query receiving module 702 described above is further configured to: determine the type of the target file of the file operation request when it is determined that the target file is accessed by the client layer; and lock multiple files in the directory when the target file is a directory.

[0246] Optionally, the query receiving module 702 described above is further configured to: determine a conflict probability threshold, wherein the conflict probability threshold is used to indicate the probability that a node layer and at least one reference node layer request a lease for the same file; and determine the lease validity period of the target file based on the conflict probability threshold, the number of client layers included in the node layer and at least one reference node layer, and the operation frequency of the target file, wherein the lease validity period is used to indicate the valid time for the client layer to hold the target lease.

[0247] For a description of the features in the embodiment corresponding to the distributed lease management device, please refer to the relevant description in the embodiment corresponding to the distributed lease management method, which will not be repeated here.

[0248] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the distributed lease management method.

[0249] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the distributed lease management method when it is run.

[0250] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0251] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the distributed lease management method.

[0252] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described distributed lease management method embodiments.

[0253] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0254] The above provides a detailed description of the distributed lease management method and system, storage medium, and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A distributed lease management method, characterized in that, For the client layer, including: Upon receiving a file operation request, a lease request is sent to the node layer, wherein the lease request is used to indicate the target lease of the target file corresponding to the file operation request; Upon receiving the target lease sent by the node layer, the target file is accessed in the storage system according to the target lease. The node layer corresponds to at least one client layer. The node layer is used to parse the file identifier and lease type of the target file corresponding to the lease request. The reference node layer refers to the node layer that holds a valid lease for the same file before the target lease request. The local lease table is checked based on the file identifier and the lease type. If the target lease conflicts with the reference node layer, the node layer sends a query request to the cluster layer. The cluster layer confirms the status of the target lease based on the query request. If the status of the target lease is released, the node layer is allowed to send the target lease.

2. The method according to claim 1, characterized in that, Sending the lease request to the node layer includes: Access the lease cache library, where the local lease cache is used to indicate at least one cached lease; If the target lease exists in the local lease cache, the target file is accessed in the storage system according to the target lease. If the target lease does not exist in the local lease cache, a lease request is sent to the node layer.

3. The method according to claim 2, characterized in that, Before accessing the target file in the storage system according to the target lease, the process includes: Upon receiving the target lease sent by the node layer, determine the operation instruction corresponding to the file operation request; When the operation instruction is a write operation, the write operation is recorded in the operation log corresponding to the node layer, wherein the operation log is used to indicate the operation type and data update information corresponding to at least one write operation.

4. The method according to claim 3, characterized in that, After accessing the target file in the storage system according to the target lease, the process includes: The status of the write operation is updated to a first status in the operation log, wherein the first status indicates that the write operation has been completed.

5. The method according to claim 2, characterized in that, Before accessing the lease cache library, the following is included: Determine the access mode corresponding to the program, and determine at least one reference file based on the access mode; Predict at least one reference lease corresponding to the program based on the at least one reference document; The at least one reference lease is cached in the lease cache library.

6. The method according to claim 3, characterized in that, After recording the write operation in the operation log corresponding to the node layer, the following steps are included: Upon restart, read the operation log; Identify at least one write operation in the second state of the operation log; Send a lease request to the node layer and perform at least one write operation.

7. A distributed lease management method, characterized in that, Used for the node layer, including: Receive a lease request sent by the client layer, wherein the lease request is used to indicate the target lease of the target file corresponding to the file operation request; Parse the file identifier and lease type of the target file corresponding to the lease request; Based on the file identifier and lease type of the target file, the local lease table is queried, and if the target lease conflicts with the reference node layer, a query request is sent to the cluster layer. The reference node layer refers to the node layer that holds a valid lease for the same file before the target lease request. The query request is used to confirm the status of the target lease. If the target lease is in a released state, the target lease is sent to the client layer.

8. The method according to claim 7, characterized in that, After querying the local lease table based on the file identifier and lease type of the target file, the process includes: If the operation instruction corresponding to the file operation request is a read operation, the target lease is sent to the client layer.

9. The method according to claim 7, characterized in that, After querying the local lease table based on the file identifier and lease type of the target file, the process includes: If the operation instruction corresponding to the file operation request is a read operation, determine the holding status of the target lease; If the holding status indicates that the target lease is not held, the target lease is sent to the client layer; If the holding status indicates that the target lease is held by the reference client layer, the file operation request is placed in the waiting queue.

10. The method according to claim 7, characterized in that, After sending the target lease to the client layer, the process includes: Update the target lease and the client layer to the local lease table; The local lease table is sent to at least one reference node layer, and the cache file of the node layer is updated based on the file operation request.

11. The method according to claim 10, characterized in that, After sending the local lease table to at least one reference node layer, the process includes: In the event that the node layer stops operating, a candidate node layer is determined from the at least one reference node layer; Based on the local lease table, at least one client corresponding to the node layer is determined, and a connection is established between the candidate node layer and the at least one client. Based on the lease request from the at least one client and the local lease table, a lease is sent to the at least one client.

12. The method according to claim 9, characterized in that, After placing the file operation request into the waiting queue, the process includes: If the target lease held by the reference client is released, determine the target lease corresponding to the file operation request; The local lease table is queried, and if the holding status indicates that the target lease is not held, the target lease is sent to the client layer.

13. A distributed lease management method, characterized in that, For the cluster layer, including: The system receives a query request from the node layer, which parses the file identifier and lease type of the target file corresponding to the lease request. Based on the file identifier and lease type, it checks the local lease table. If the target lease conflicts with the reference node layer, the node layer sends a query request to the cluster layer. The reference node layer refers to the node layer that held a valid lease for the same file before the target lease request. The query request is used to query the status of the target lease, which is the lease of the target file corresponding to the file operation request processed by the client layer. The global lease index is determined based on the query request, wherein the global lease index is used to indicate the local lease tables corresponding to the node layer and at least one reference node layer; If the target lease is in a released state, an authorization instruction is sent to the node layer, wherein the authorization instruction is used to instruct the node layer to send the target lease to the client layer if the node layer receives the authorization instruction sent by the cluster layer.

14. The method according to claim 13, characterized in that, When the target lease is in a released state, sending an authorization instruction to the node layer includes: Send a coordination instruction to the reference node layer that holds the target lease; If the target lease is released at the reference node layer or the holding time of the target lease exceeds a preset time, an authorization instruction is sent to the node layer.

15. The method according to claim 13, characterized in that, Before the query request is sent by the receiving node layer, the following is included: If it is determined that the client layer accesses the target file, the type of the target file in the file operation request is determined; If the target file is a directory, lock multiple files within that directory.

16. The method according to claim 13, characterized in that, Before the query request is sent by the receiving node layer, the following is included: Determine a conflict probability threshold, wherein the conflict probability threshold is used to indicate the probability that the node layer and at least one reference node layer request a lease for the same file; The lease validity period of the target file is determined based on the conflict probability threshold, the number of client layers included in the node layer and at least one reference node layer, and the operation frequency of the target file, wherein the lease validity period is used to indicate the effective time for the client layer to hold the target lease.

17. A distributed lease management system, characterized in that, include: The client layer is used to perform the method described in any one of claims 1 to 6; A node layer, used to perform the method described in any one of claims 7 to 12; A cluster layer for performing the method according to any one of claims 13 to 16; A storage device used to store multiple files, including the target file.

18. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the distributed lease management method as described in any one of claims 1 to 16 when executing the computer program.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the distributed lease management method as described in any one of claims 1 to 16.

20. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the distributed lease management method as described in any one of claims 1 to 16.

Images