Locking methods and devices for locking directory trees

Abstract

This invention provides a locking method for locking a directory tree and related products. The locking method includes: obtaining an operation request; and performing MGL operations on multiple MGL lock objects in a multiple-granularity locking (MGL) lock object array according to the operation request. The MGL lock object array includes multiple MGL lock objects used to record the locking status of nodes in the directory tree. Each MGL lock object corresponds to multiple nodes, and each node includes at least one target node and one or more parent nodes. This method enables MGL for the directory tree, thereby improving the concurrency of file system metadata tree operations.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a locking method and locking device for locking a directory tree. Background Technology

[0002] The hierarchical structure of files and directories in a file system can be represented as a tree structure, i.e., a directory tree. The root node in a directory tree represents the root directory " / ", each intermediate node represents a subdirectory, and each leaf node represents a subdirectory or file.

[0003] Read-write locks are the most commonly used structure in related technologies. They are used to synchronize file system operations and prevent the current operation from being affected by other operations. If an operation needs to read the contents of a node, it needs to acquire a read lock; if an operation needs to modify the contents of a node, it needs to acquire a write lock. Read-write locks can ensure that concurrent operations are not interfered with, but their concurrency is relatively poor.

[0004] The purpose of the background art is to disclose information that the applicant believes may be relevant to the present invention. It is neither necessary nor appropriate to acknowledge that any of the foregoing information constitutes prior art in relation to the present invention. Summary of the Invention

[0005] In a first aspect, one embodiment of the present invention provides a locking method for locking a directory tree, implemented by a locking device. The locking method includes:

[0006] Get the operation request;

[0007] According to the operation request, an MGL operation is performed on multiple MGL lock objects in a multiple-granularity locking (MGL) lock object array, wherein the MGL lock object array includes multiple MGL lock objects for recording the locking status of nodes in the directory tree, the multiple MGL lock objects correspond to multiple nodes in the directory tree, and the multiple nodes include at least one target node of the operation request and one or more parent nodes of the at least one target node.

[0008] By using an array of MGL lock objects to record the locking status of nodes in the directory, and performing MGL operations on the MGL lock objects corresponding to nodes in the directory tree (including at least one target node affected by the operation request and one or more parent nodes of the at least one target node), MGL locks of different granularities can be set according to the operation request. Therefore, MGL of the directory tree can be implemented, which allows concurrent operations to be performed without conflict, thereby improving the concurrency of file system metadata tree operations.

[0009] In one possible implementation of the first aspect, each node in the directory tree corresponds to a corresponding path in the file system, and each of the plurality of nodes is mapped to a corresponding MGL lock object based on the path information corresponding to the corresponding node. Since the nodes in the directory tree correspond to paths in the file system, each node can be mapped to a corresponding MGL lock object based on its corresponding path information. That is, the index of the MGL lock object mapped to by a node can be determined based on the path information of that node. In this way, the mapping between nodes in the directory tree and MGL lock objects can be easily and efficiently implemented.

[0010] In one possible implementation of the first aspect, each of the plurality of nodes is mapped to the corresponding MGL lock object according to the hash code value of the path corresponding to the corresponding node.

[0011] In one possible implementation of the first aspect, the hash code value is obtained by using a hash algorithm on at least one string associated with the path.

[0012] Since each node corresponds to a specific path, the hash code value of that path can be used to map the node to the corresponding MGL lock object. Paths are typically represented by strings, and the hash code value of a path can be determined by using a hash algorithm on at least one string associated with that path. This provides a possible and efficient way to map nodes to MGL lock objects.

[0013] In one possible implementation of the first aspect, the MGL lock object array comprises N MGL lock objects, where N is an integer greater than 1.

[0014] The size N of the MGL lock object array can be an integer greater than 1, which can be set according to actual conditions or needs, thereby improving the adaptability and flexibility of the technical solution.

[0015] In one possible implementation of the first aspect, each of the N MGL lock objects in the MGL lock object array has a unique index.

[0016] Because each MGL lock object has a unique index, nodes in the directory tree can be accurately mapped to that MGL lock object.

[0017] In one possible implementation of the first aspect, the indices of the N MGL lock objects are in the range [0, N–1]. In another possible implementation of the first aspect, each of the plurality of nodes is mapped to the corresponding MGL lock object based on the remainder obtained by dividing the hash code value of the path corresponding to the corresponding node by N.

[0018] When the size of the MGL lock object array is N, the index range of the MGL lock objects in the array can be from 0 to N–1. A modulo-N operation can be performed on the hash code value of the path to map the hash code value to a number in the range [0, N–1], thereby mapping each node to the corresponding MGL lock object in the array.

[0019] In one possible implementation of the first aspect, performing an MGL operation on a plurality of MGL lock objects in the MGL lock object array according to the operation request includes:

[0020] Locks are applied to the plurality of MGL lock objects according to the operation request, wherein the lock mode of the lock to be applied to the MGL lock object is determined according to the operation request.

[0021] By applying locks to MGL lock objects corresponding to one or more target nodes and one or more parent nodes according to the operation request, these nodes can be locked in a mode suitable for the operation request, thereby preventing the operation request from being interfered with by one or more other operation requests.

[0022] In one possible implementation of the first aspect, if a lock has already been applied to the first MGL lock object among the plurality of MGL lock objects, and the lock mode of the lock to be applied to the first MGL lock object conflicts with the lock mode of the lock already applied to the first MGL lock object, then the plurality of MGL lock objects are not locked according to the operation request before the lock already applied to the first MGL is released.

[0023] When a lock to be applied to an MGL lock object conflicts with a lock already applied to an MGL lock object, the operation request will not apply locks to multiple MGL lock objects until the lock already applied to the MGL lock object is released, thereby avoiding interference with one or more other operation requests in the process.

[0024] In one possible implementation of the first aspect, if the operation request indicates an operation affecting a target node, and the operation is a read operation, then a shared (S) lock is applied to the MGL lock object corresponding to the affected target node, and an intention shared (IS) lock is applied to each of the one or more MGL lock objects corresponding to one or more parent nodes of the affected target node.

[0025] When an operation request indicates an operation that affects a target node and that operation is a read operation, an S lock is applied to the MGL lock object corresponding to the affected target node, and an IS lock is applied to one or more MGL lock objects corresponding to one or more parent nodes. Because locks are applied to nodes at different levels, while IS locks are applied to one or more MGL lock objects corresponding to one or more parent nodes, more compatible operations can be processed simultaneously, thereby improving the concurrency of the file system.

[0026] In one possible implementation of the first aspect, if the operation request indicates an operation affecting a target node, and the operation is a write operation, then an exclusive (X) lock is applied to the MGL lock object corresponding to the affected target node, and an intention exclusive (IX) lock is applied to each of the one or more MGL lock objects corresponding to one or more parent nodes of the affected target node.

[0027] When an operation request indicates an operation that affects a target node and that operation is a write operation, an X lock is applied to the MGL lock object corresponding to the affected target node, and an IX lock is applied to one or more MGL lock objects corresponding to one or more parent nodes. Because IX locks are applied to one or more MGL lock objects corresponding to one or more parent nodes, more compatible operations can be processed simultaneously, thus improving file system concurrency.

[0028] In one possible implementation of the first aspect, if the operation request indicates an operation affecting two or more target nodes, the plurality of MGL lock objects include one or more second MGL lock objects that each require one lock mode and one or more third MGL lock objects that each require two or more lock modes. The lock mode of the lock to be applied to each of the one or more second MGL lock objects is determined to be the lock mode required by the corresponding second MGL lock object, and the lock mode of the lock to be applied to each of the one or more third MGL lock objects is determined based on the most stringent lock mode among the two or more lock modes required by the corresponding third MGL lock object.

[0029] When an operation request indicates an operation affecting two or more target nodes, there can be one or more MGL lock objects, each requiring only one lock mode, or one or more other MGL lock objects, each requiring two or more lock modes. For an MGL lock object requiring only one lock mode, the required lock mode is considered the lock mode of the lock to be applied to that MGL lock object. For an MGL lock object requiring two or more lock modes, the most stringent lock mode among the two or more lock modes is considered the lock mode of the lock to be applied to that MGL lock object. In this way, complex operations affecting two or more target nodes can also be implemented using MGL, and complex operations can be executed concurrently with other compatible operations, thereby further improving the concurrency of the file system.

[0030] In one possible implementation of the first aspect, if the target node and one or more parent nodes of the target node include two or more nodes mapped to the same MGL lock object, then the lock mode of the lock to be applied to the same MGL lock object is determined according to the lock modes required by the two or more nodes respectively.

[0031] When two or more nodes on a path are mapped to the same MGL lock object, the lock mode to be applied to that MGL lock object can be determined based on all the lock modes required by the two or more nodes, thereby avoiding interference from one or more other operation requests.

[0032] In a second aspect, one embodiment of the invention provides a locking device. The locking device includes various modules for performing the locking method for locking a directory tree according to the first aspect or any possible implementation thereof.

[0033] In a third aspect, one embodiment of the invention provides a locking device. The locking device includes processing circuitry for performing the locking method for locking a directory tree according to the first aspect or any possible implementation thereof.

[0034] In a fourth aspect, one embodiment of the present invention provides a file system. The file system includes a locking device according to either the second or third aspect.

[0035] In a fifth aspect, one embodiment of the invention provides a file system. The file system includes a metadata management component for performing the locking method for locking a directory tree as described in the first aspect or any possible implementation thereof.

[0036] In a sixth aspect, one embodiment of the invention provides a computer-readable medium storing computer-executable instructions. When executed by a processor, the computer-executable instructions cause the processor to perform a locking method for locking a directory tree according to the first aspect or any possible implementation thereof.

[0037] In a seventh aspect, one embodiment of the invention provides a computer program product including computer-executable instructions. When executed by a processor, the computer-executable instructions cause the processor to perform a locking method for locking a directory tree according to the first aspect or any possible implementation thereof.

[0038] This invention provides a locking method and related products for locking a directory tree. An array of MGL lock objects is used to record the lock states of nodes in the directory tree. When an operation request is received, an MGL operation is performed on the MGL lock objects corresponding to one or more target nodes affected by the operation request and one or more parent nodes of the target nodes, according to the operation request. In this way, MGL locks of different granularities can be set according to the operation request, thus enabling MGL of the directory tree. This allows concurrent operations to be performed without conflict, thereby improving the concurrency of file system metadata tree operations. Attached Figure Description

[0039] The following figures illustrate exemplary embodiments of the present invention by way of example.

[0040] Figure 1 This is a schematic diagram of the HDFS architecture.

[0041] Figure 2 This is a schematic diagram of the NameNode structure.

[0042] Figure 3This is a diagram illustrating the NameNode's workflow.

[0043] Figure 4 This is a schematic flowchart of a locking method for locking a directory tree provided by one or more exemplary embodiments of the present invention.

[0044] Figure 5 This is a schematic diagram of a lock for read operations provided by one or more possible embodiments of the present invention.

[0045] Figure 6 This is a schematic diagram of a lock for changing operations provided by one or more possible embodiments of the present invention.

[0046] Figure 7 This is a schematic diagram illustrating an example of concurrent operation provided by one or more possible embodiments of the present invention.

[0047] Figure 8 This is a schematic diagram of a lock for snapshot operations provided by one or more possible embodiments of the present invention.

[0048] Figure 9 This is a schematic diagram illustrating an example of a conflict scenario provided by one or more possible embodiments of the present invention.

[0049] Figure 10 This is a schematic diagram of the structure of a locking device provided by one or more exemplary embodiments of the present invention. Detailed Implementation

[0050] In the following description, reference is made to the accompanying drawings, which form part of this invention, which illustrate by way of description specific aspects of embodiments of the invention or aspects in which embodiments of the invention may be used. It should be understood that embodiments of the invention can be used in other aspects and include structural or logical variations not depicted in the drawings. Therefore, the following detailed description should not be construed as limiting, and the scope of the invention is defined by the appended claims.

[0051] A file system is a method of organizing files and directories on storage media. A distributed file system (DFS) is a file system distributed across multiple servers. DFS allows users to share files and storage over a network and is widely used due to its high availability and scalability. The hierarchical structure of files and directories in a file system or DFS can be represented as a tree structure, i.e., a directory tree. The root node in the directory tree represents the root directory " / ", each intermediate node represents a subdirectory, and each leaf node represents a subdirectory or file. The ability to handle concurrent operations is crucial to the performance of the entire system, and DFS may be accessed by a large number of users or applications simultaneously, thus requiring particularly good concurrency.

[0052] Read-write locks are the most commonly used structure in related technologies. They are used to synchronize file system operations and prevent the current operation from being affected by other operations. If an operation needs to read the contents of a node, it needs to acquire a read lock; if an operation needs to modify the contents of a node, it needs to acquire a write lock. Read-write locks ensure that concurrent operations are not disturbed.

[0053] In existing technical solutions, the entire metadata tree uses a single read-write lock. This approach is simple to implement, but suffers from poor concurrency because each metadata tree operation locks the entire metadata tree, preventing other operations from being performed simultaneously.

[0054] In another technical approach, read-write locks are added to each node in the metadata tree involved in the operation. Specifically, a fixed-size array of read-write locks is applied to each level of the metadata tree. The canonical path of the destination object is divided into directories, and locks at each level are applied based on the directory name. The locking pattern depends on the requested operation. For any read operation, read locks are applied to all nodes involved in the operation, while for any create / delete operation, write locks are applied to the parent node and read locks are applied to other higher-level nodes. This approach offers better concurrency compared to semaphore read-write locks. However, for operations that modify directory contents, write locks are applied to the parent node, thus blocking any other parallel operations within the directory. Furthermore, this approach is not practical for real-world applications because it significantly increases the amount of random access memory (RAM) used in the metadata tree.

[0055] This invention provides a locking technique that improves the performance of parallel operations on a file system metadata tree stored in memory by using a set of multi-granularity locks. The technical solution is illustrated below using the Hadoop Distributed File System (HDFS) as an example. However, this technique is applicable to any file system that stores the working dataset in a tree structure in memory and performs concurrent modifications on different subtrees. Generally, as long as the metadata tree is in memory, this technique can be used to synchronize concurrent metadata tree operations on any distributed or local file system.

[0056] To facilitate understanding of the technical solution of this invention, some terms mentioned in this invention will be introduced first.

[0057] Multiple granularity locking (MGL) is a locking method that sets a lock on objects that contain other objects. MGL leverages the hierarchy of containment relationships. For example, a database can include files, files can include pages, and pages can include records. This can be viewed as an object tree where each node includes its child nodes. A lock on this structure (e.g., a shared lock or an exclusive lock) locks the target node and all its descendant nodes.

[0058] The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on general-purpose hardware. It shares many similarities with existing distributed file systems.

[0059] HDFS is highly fault-tolerant and designed for deployment on low-cost hardware.

[0060] HDFS provides high-throughput access to application data and is suitable for applications with large datasets.

[0061] HDFS relaxes some of the requirements of the Portable Operating System Interface (POSIX) standard to enable streaming access to file system data.

[0062] HDFS was originally built as the infrastructure for the Apache Nutch web search engine project.

[0063] HDFS is part of the Apache Hadoop Core project.

[0064] The HDFS NameNode is a master server that manages the file system namespace, controls client access to files, and performs file system namespace operations, such as opening, closing, and renaming files and directories. It also determines the mapping of file data blocks to DataNodes. The simplest HDFS configuration consists of one NameNode instance and several DataNodes.

[0065] HDFS DataNodes manage storage connected to the nodes on which the DataNodes run. HDFS exposes a file system namespace and allows user data to be stored in files. Internally, files are divided into one or more blocks, which are stored across a set of DataNodes.

[0066] DataNode is responsible for handling read and write requests from file system clients.

[0067] DataNode also performs block creation, deletion, and replication based on the NameNode's instructions.

[0068] MapReduce is a programming paradigm that enables massive scaling across hundreds or thousands of servers in a Hadoop cluster. As a processing component, MapReduce is the core of Apache Hadoop. The term "MapReduce" refers to two independent and distinct tasks executed by a Hadoop program.

[0069] An INode is a data structure in a Unix-style file system that describes file system objects, such as files or directories.

[0070] An INode ID is a unique identifier for an INode object.

[0071] Data blocks are fixed-size chunks in which HDFS stores file content.

[0072] FsImage is a file stored in the operating system (OS) file system. This file contains the complete directory structure (namespace) of HDFS, as well as details on the location of data on data blocks and which blocks are stored on which DataNode.

[0073] EditLog is the transaction log that records changes in the HDFS file system or any operations performed on the HDFS cluster, such as adding new blocks, copying, deleting, etc. EditLog includes changes since the creation of the last FsImage.

[0074] BlockMap is a mapping that maps blocks in a file to the physical locations of those blocks in a DataNode.

[0075] INodeMap is a mapping between INode IDs and INode objects.

[0076] A canonical path is a path to a file or a directory starting from the root of the file system. It does not include dot (.), double dot (..) components, nor does it include any repeating separators ( / ), symbolic links, or reserved segments like ".reserved".

[0077] Apache Hadoop is an open-source framework for storing and processing large datasets ranging in size from gigabytes to petabytes. Hadoop is an Apache project, and all components are available under the Apache Open Source License. The main module of Hadoop is the Distributed File System (HDFS), which was originally designed to run on standard or low-end hardware. In addition to high fault tolerance and native support for large datasets, HDFS offers better data throughput than traditional file systems. HDFS stores file system metadata and application data separately. A dedicated server (called the NameNode) is responsible for metadata storage and processing. Application data is stored on other servers (called DataNodes). All servers are fully connected and communicate with each other via a TCP-based protocol. Applications can access HDFS in many different ways. Natively, HDFS provides a Java API for application use. A C wrapper is also provided for the Java API. Additionally, the HTTP protocol can be used to browse files within an HDFS instance.

[0078] Figure 1 This is a schematic diagram of the HDFS architecture. (For example...) Figure 1 As shown, HDFS consists of the following components:

[0079] NameNode:

[0080] • Manage file system namespaces and control client access to files.

[0081] • Perform file system namespace operations, such as opening, closing, and renaming files and directories.

[0082] • Determine the mapping from file data blocks to DataNodes

[0083] DataNode:

[0084] • Manage the storage connected to the nodes where they run.

[0085] • Responsible for providing services for read and write requests from file system clients.

[0086] • Execute block creation, deletion, and replication according to the NameNode's instructions.

[0087] Figure 2This is a schematic diagram of the NameNode structure. The HDFS namespace is a hierarchical structure of files and directories. Files and directories are represented by INodes on the NameNode. INodes record attributes such as permissions, modification and access counts, namespace and disk space quotas, and links to child INodes. File content is divided into large blocks (typically 128 megabytes, but users can select them file by file), and each block in a file is stored on one or more DataNodes. The NameNode maintains the namespace tree and the mapping of blocks to DataNodes.

[0088] Figure 3 This diagram illustrates the NameNode's workflow, where multiple processors use a single global lock to handle requests and write changes to the EditLog file. Currently, in the open-source version of Hadoop, namespaces are stored in a thread-unsafe tree, and all operations are performed under a global read-write lock. This becomes a bottleneck when there are many concurrent client requests. Performance can be improved by replacing the combination of the thread-unsafe tree and the global read-write lock, and by using techniques that allow concurrent operations to be performed on different nodes without conflict.

[0089] Figure 4 This is a schematic flowchart illustrating a method for locking a directory tree according to one or more exemplary embodiments of the present invention. The locking method can be implemented by a locking device. Optionally, the locking device can be a file system metadata management component that stores all metadata in memory, such as the NameNode in HDFS, or other components with similar functionality, which are not limited herein. Figure 4 As shown, the locking method may include the following steps.

[0090] S410: Obtain operation request;

[0091] An operation request can be obtained, which indicates the operation to be performed. Each node in the directory tree corresponds to a path in the file system. An operation request may affect one or more target nodes in the directory tree. For example, if the operation request indicates reading the contents of / dir1 / dir2 / , where dir1 and dir2 are directory names, then the target node affected by the operation request is / dir1 / dir2. As another example, if the operation request indicates creating a new file / dir1 / dir2 / readme.txt, a new node, / dir1 / dir2 / readme.txt, will be created, and the target node will be the newly created node. Yet another example, if the operation request indicates moving the file / dir1 / dir2 / readme.txt to a different folder / dir1 / dir3, the node / dir1 / dir2 / readme.txt will be removed, and a new node / dir1 / dir3 / readme.txt will be created. In this example, both the old node / dir1 / dir2 / readme.txt and the new node / dir1 / dir3 / readme.txt are affected by the operation request, and there are two target nodes. Such operations that affect two or more target nodes can also be referred to as "complex operations" in this invention.

[0092] S420: Perform an MGL operation on multiple MGL lock objects in a multiple-granularity locking (MGL) lock object array according to the operation request. The MGL lock object array includes multiple MGL lock objects used to record the locking status of nodes in the directory tree. The multiple MGL lock objects correspond to multiple nodes in the directory tree. The multiple nodes include at least one target node of the operation request and one or more parent nodes of the at least one target node.

[0093] An array of MGL lock objects is used to lock state nodes in a directory tree. In other words, an array of MGL lock objects is used to apply locks to nodes in a directory tree or paths in a file system. Since the height of a file system hierarchy (consisting of folders, subfolders, and files) can be virtually unlimited, the size of the MGL lock object array and the mapping between nodes and MGL lock objects are critical to the implementation of the technical solution.

[0094] The locking method for locking a directory tree provided in this invention can improve the throughput of concurrent HDFS NameNode file operations without conflicting with each other, provided that the operation paths and memory usage are consistent. For example, it can simultaneously perform operations to create different files in the same directory, simultaneously perform operations to read the contents of any directory, simultaneously perform operations to move directory A and read the contents of directory B (these two directories have different paths or a common parent directory), or perform directory snapshots by copying the contents of a directory to a subdirectory.

[0095] By replacing a global read-write lock with the MGL lock object described in this invention, the technical solution provided by the embodiments of this invention can be implemented as an improvement to the Hadoop NameNode. Furthermore, these technical solutions can also be applied to other programs that perform concurrent read / modification operations on tree structures.

[0096] In one possible implementation, the MGL lock object array comprises N MGL lock objects, where N is an integer greater than 1. In another possible implementation, the size N of the MGL lock object array is fixed. When using a fixed-size MGL lock object array, the RAM usage of this array is also fixed and does not increase with the height of nodes or the directory tree.

[0097] In one possible implementation, the size N of the MGL lock object array can be configured and initialized at file system startup. The size N can be configured according to actual conditions or requirements, improving the adaptability and flexibility of the technical solution.

[0098] If the value of N is very small, the likelihood of lock contention (where two or more nodes map to the same MGL lock object) is high. While MGL can still be used in the event of lock contention, as described below, frequent contention can negatively impact system performance. In one possible implementation, the value of N is between one hundred and several thousand.

[0099] In one possible implementation, the value of N can be configured based on the specifications of one or more machines on which the system runs. In another, the size N can be configured based on the number of central processing unit (CPU) cores. When the number of MGL lock objects is small compared to the number of cores, the system throughput is linearly limited by the number of MGL lock objects. On the other hand, when the number of MGL lock objects is very large, bandwidth slowly reaches its theoretical maximum, but the lock array consumes excessive memory. For example, if N equals the number of CPU cores, concurrency might average around 63% of the maximum due to conflicts; if N is twice the number of CPU cores, concurrency might average around 79%; if N is three times the number of CPU cores, concurrency might average around 85%; as the number of locks increases further, the concurrency gain becomes negligible. Additionally, it is worth considering that if the CPU workload occupied by the critical section is small, it is acceptable to proportionally reduce the number of MGL lock objects. In one possible implementation, N can be twice the number of CPU cores, providing a high starting point. For example, on a server with 64 cores, N=128 could be used.

[0100] In addition to the shared (S) mode (also known as read mode) and exclusive (X) mode (also known as write mode), MGL provides three additional lock modes: intention-shared (IS) mode, which indicates explicit locking at a lower level of the tree using one or more shared locks; intention-exclusive (IX) mode, which indicates explicit locking at a lower level using one or more exclusive locks and / or one or more shared locks; and shared and intention-exclusive (SIX) mode, which indicates explicit locking of the subtree rooted at the node in shared mode, with the explicit locking performed at a lower level using at least one exclusive lock.

[0101] An MGL operation is performed on multiple MGL lock objects corresponding to multiple nodes involved in an operation request. The multiple nodes involved in the operation request include at least one target node of the operation request and one or more parent nodes of each target node, wherein the one or more parent nodes of each target node include all or more parent nodes of the target node up to the root. These nodes can be mapped to MGL lock objects based on some unique information about these nodes (such as path information of the corresponding nodes).

[0102] By using an array of MGL lock objects to record the locking status of nodes in the directory, and performing MGL operations on the MGL lock objects corresponding to nodes in the directory tree (including at least one target node affected by the operation request and one or more parent nodes of the at least one target node), MGL locks of different granularities can be set according to the operation request. Therefore, MGL can be implemented on the directory tree in the file system, which allows concurrent operations to be performed without conflict, thereby improving the concurrency of file system metadata tree operations.

[0103] In one possible implementation, each node in the directory tree corresponds to a corresponding path in the file system, and each of the multiple nodes is mapped to a corresponding MGL lock object based on the path information corresponding to that node. Since each node in the directory tree corresponds to a path in the file system, each node can be mapped to a corresponding MGL lock object based on its corresponding path information. That is, the index of the MGL lock object to which a node is mapped can be determined based on the path information corresponding to that node. Because the file system metadata management component knows the path corresponding to each node, using path information for mapping can more easily and efficiently implement the mapping between nodes in the directory tree and MGL lock objects, without needing to allocate additional identification information for each node or obtain physical address information for mapping, which might require more computational and storage resources.

[0104] In one possible implementation, each of the multiple nodes is mapped to a corresponding MGL lock object based on the hash code value of the path corresponding to that node. Each directory or file can be named using a string of characters according to certain naming rules, where these characters can include letters, numbers, and even special characters such as "-" and "_". The path to the target directory can be represented by a string, including the names of a series of directories from the root directory to the target directory; the path to the target file can be represented by a string, including the names of a series of directories from the root directory to the file's parent directory, and the file name. Since paths are typically represented by strings of characters, and the index of an MGL lock object is an integer, a hash can be used to convert the string or other information derived from the string into an integer value, from which the index of the MGL lock object can be obtained. Therefore, a hash code value can be derived from the string, and then used to map a node or path to the corresponding MGL lock object. Because the hash code values ​​for the same path are identical, the mapping between nodes and MGL lock objects remains unchanged (unless a new node is created or an old node is removed). The same node can always be mapped to the same MGL lock object, allowing the same MGL lock object to accurately record the locking state of the same node. This way, when an operation request is received, the current locking state of the relevant node can be accurately obtained, thus facilitating the implementation of MGL.

[0105] In one possible implementation, the hash code value can be obtained by hashing at least one string associated with the path, providing a possible and efficient way to map nodes to MGL lock objects. In another possible implementation, standard Java hashCode calculation can be used to obtain the hash code value of the path. The input to the hashCode calculation can be the entire string representing the path. Alternatively, the path can be represented as a segmented array of strings, starting from the root (an empty string), and then the hash code value of the string array can be calculated using Java static methods such as Arrays.hashCode(String[]). For example, for the path / foo / bar, the hash code value of the array {"", "foo", "bar"} consisting of three elements can be obtained. Using a segmented array to obtain the hash code can reduce computation or improve computational efficiency. Specifically, in the technical solution provided by this invention, it is necessary not only to calculate the hash code of the target node, but also to calculate the hash codes of all parent nodes of the target node starting from the root. In the process of calculating the hash code of the segmented array, the calculation is performed from left to right along the array. As an intermediate result, the hash code values ​​of the paths of all parent nodes starting from the root can also be obtained. In this way, it is not necessary to perform two unnecessary and cumbersome operations: obtaining the paths of all parent nodes from the path of the target node and calculating the hash code value of each parent node respectively.

[0106] In one possible implementation, each of the N MGL lock objects in the MGL lock object array has a unique index. Because each MGL lock object has a unique index, each node in the directory tree can be accurately mapped to the corresponding MGL lock object.

[0107] In one possible implementation, the indices of the N MGL lock objects are in the range [0, N–1]. In another possible implementation, each node among multiple nodes can be mapped to its corresponding MGL lock object based on the remainder obtained by dividing the hash code value of the path corresponding to that node by N. That is, when the size of the MGL lock object array is N, the indices of the MGL lock objects in the array can range from 0 to N–1, and a modulo N operation can be performed on the hash code value of the path to map the hash code value to an integer in the range [0, N–1], thereby mapping each node to its corresponding MGL lock object in the array. In this way, an MGL lock object array of size N can be used to lock the directory tree. The indices of the MGL lock objects can have other values, and nodes in the directory tree can also be mapped to MGL lock objects in other ways.

[0108] Taking the path / foo / bar as an example again, the hash code of the array {"", "foo", "bar"} consisting of three elements is 3275884. If N is set to 50, the index of the MGL lock object can be calculated as 3275884 mod 50 = 34. Therefore, the node corresponding to the path / foo / bar can be mapped to the MGL lock with index 34.

[0109] For example, for the canonical path / dir1 / dir2 / ... / dir M–1 / dir M The corresponding target node, whose parent node corresponds to the path from / to / dir1 / dir2 / ... / dir M–1 The mapping of subpaths, target nodes, and parent nodes to MGL lock objects can be performed as follows:

[0110] H0 = hashCode(" / ") mod N

[0111] H1 = hashCode(" / dir1") mod N

[0112] H2 = hashCode(" / dir1 / dir2") mod N

[0113] ...

[0114] H M–1 = hashCode(" / dir1 / dir2 / … / dir M–1 mod N

[0115] H M = hashCode(" / dir1 / dir2 / … / dir M–1 / dir M mod N

[0116] Here, `mod` represents the modulo operation, and `hashCode(s)` represents the hash function, which returns the hash code value of a given input string `s`. The input string `s` can be the entire string representing the path, or it can be a segmented array of strings representing the path, as described above. For example, when using the entire string as the input string, the hash code of the target node ` / dir1 / dir2 / ... / dir` is calculated separately for each string as input. M–1 / dir M and from / to / dir1 / dir2 / ... / dir M–1 The hash code of each parent node in the parent node. For example, when using a segmented array, the array {"", "dir1", "dir2",... , "dirM The string "} is the input string for calculating the hash code value of the target node. This string, used as an intermediate result, allows us to obtain the hash codes from the root ` / ` to ` / dir1 / dir2 / ... / dir`. M–1 The hash codes of all parent node paths are used to calculate the final result of the hash codes for the target node / dir1 / dir2 / ... / dir M–1 / dir M The hash code value. Values ​​H0…H M Sort the MGL lock objects in ascending order to represent the MGL lock objects for which MGL operations will be applied. This sorting is done to prevent deadlocks during concurrent operations.

[0117] In one possible implementation, performing MGL operations on multiple MGL lock objects in an MGL lock object array based on an operation request is done by applying locks to the multiple MGL lock objects according to the operation request, wherein the lock mode of the lock to be applied to the MGL lock object is determined based on the operation request. By applying locks to the MGL lock objects corresponding to one or more target nodes and one or more parent nodes according to the operation request, these nodes can be locked in a mode suitable for the operation request, thereby preventing the operation request from being interfered with by one or more other operation requests.

[0118] An operation may affect one or more target nodes. Operations affecting a target node can be divided into read operations and write operations. A read operation refers to an operation that does not change the local state of the affected target node, while a write operation refers to an operation that changes the local state of the affected target node. The local state of the affected target node can include the existence of the affected target node, the content of the affected target node, and the configurable attributes of the affected target node.

[0119] The existence of a node can refer to its state of being present or absent. For example, creating the file ` / dir1 / dir2 / readme.txt` results in the creation of a new node ` / dir1 / dir2 / readme.txt`. The node ` / dir1 / dir2 / readme.txt` changes from non-existent to existing; its existence is altered by the creation operation, therefore, creation is a write operation. Similarly, deleting the directory ` / dir1 / dir3` results in the deletion of the node ` / dir1 / dir3`. The node ` / dir1 / dir3` changes from existing to non-existent; its existence is altered by the deletion operation, therefore, deletion is also a write operation. However, for the operation of reading the file ` / dir1 / dir2 / readme.txt`, the node ` / dir1 / dir2 / readme.txt` exists both before and after the operation; its existence is not changed by this operation.

[0120] Configurable properties refer to attributes that applications or users can configure individually. Generally, these properties do not change automatically with changes to the contents of a file or directory, nor do they change due to read operations. For files, configurable properties may include the file's location, file permissions, and file replication factor (i.e., the number of copies). For directories, configurable properties may include the directory's location, directory permissions, and directory quotas. Directory quotas may include name quotas for the number of names used and space quotas for the amount of space used by the directory.

[0121] An operation is a read operation if it does not change the local state of the affected target node—for example, it does not create or delete the node, nor does it change the content or configurable properties of the affected target node. Read operations can include reading directory contents, reading file contents, opening files, and so on.

[0122] An operation is a write operation if it changes the local state of an affected target node—for example, if the operation creates or deletes the target node, or changes the content or configurable properties of the affected target node. Write operations can include creating files or directories, deleting files or directories, updating files, changing file or directory permissions, changing file replication factors, changing directory quotas, and so on.

[0123] In one possible implementation, the operation request may indicate an operation affecting a target node, which is a read operation. In this case, an S lock is applied to the MGL lock object corresponding to the target node, and an IS lock is applied to each of the one or more MGL lock objects corresponding to one or more parent nodes of the target node.

[0124] When an operation request indicates an operation that affects a target node and that operation is a read operation, an S lock can be applied to the MGL lock object corresponding to the affected target node, and an IS lock can be applied to each of the one or more MGL lock objects corresponding to one or more affected parent nodes. For example, for reading / dir1 / dir2 / … / dir M–1 / dir M Operations on the content from / to / dir1 / dir2 / ... / dir M–1 / dir M Apply IS locks to all MGL lock objects corresponding to the subpaths, and apply IS locks to / dir1 / dir2 / ... / dir M–1 / dir MAn S lock is applied to the corresponding MGL lock object. For example, when the operation is to read ` / dir1 / dir3 / readme.txt`, an S lock is applied to the MGL lock object corresponding to the target node ` / dir1 / dir3 / readme.txt`, and IS locks are applied to the MGL lock objects corresponding to the three parent nodes ` / `, ` / dir1`, and ` / dir1 / dir3`. Because locks are applied to nodes at different levels, while IS locks are applied to one or more MGL lock objects corresponding to one or more parent nodes, more compatible operations can be processed simultaneously, thus improving the concurrency of the file system.

[0125] Figure 5 This is a schematic diagram of a lock for read operations provided by one or more possible embodiments of the present invention. Figure 5 This demonstrates how to apply a lock to an MGL lock object when an operation request instructs the reader to read the contents of / dir1 / dir2. For example... Figure 5 As shown in 530, the target nodes / dir1 / dir2 are mapped to the MGL lock object Lock. N As shown in 520, the parent node / dir1 maps to the MGL lock object Lock. N–2 As shown in 510, another parent node (i.e., the root node / ) is mapped to the MGL lock object Lock1. Since this operation is a read operation, the MGL lock object Lock corresponding to the target node / dir1 / dir2 is also mapped. N Apply an S-lock, and apply the MGL lock objects Lock1 and Lock2 corresponding to the two parent nodes / and / dir1. N–2 Apply IS locks separately. In this case, since / dir1 / dir2 is locked in S mode, while / and / dir1 are locked in IS mode, therefore, except for the three directories or other subdirectories under these three directories ( Figure 5 Read operations can be performed in directories (not shown in the image), and also in other directories such as / dir1, / dir1 / dir3, etc. under / dir1. Figure 5 (not shown in the image) and other directories such as / dir4 and / dir5 under the root path / (not shown in the image) Figure 5 (Not shown in the image) Simultaneously performing some write operations, such as creating files or directories, or deleting files, allows for more concurrent operations, thus improving the file system's concurrency. This approach also applies to other read operations.

[0126] In one possible implementation, the operation request may indicate an operation affecting a target node, which is a write operation. In this case, an X lock is applied to the MGL lock object corresponding to the target node, and an IX lock is applied to each of the one or more MGL lock objects corresponding to one or more parent nodes of the target node.

[0127] When an operation request indicates an operation that affects a target node and that operation is a write operation, an X lock is applied to the MGL lock object corresponding to the target node, and an IX lock is applied to each of the one or more MGL lock objects corresponding to one or more parent nodes. For example, for creating / dir1 / dir2 / … / dir M–1 / dir M Operations on / to / dir1 / dir2 / ... / dir M–1 / dir M–1 Apply IX locks to all MGL lock objects corresponding to the subpaths, and apply them to / dir1 / dir2 / ... / dir M–1 / dir M An X lock is applied to the corresponding MGL lock object. For example, when the operation is to delete ` / dir1 / dir3 / readme.txt`, an X lock is applied to the MGL lock object corresponding to the target node ` / dir1 / dir3 / readme.txt`, and IX locks are applied to the MGL lock objects corresponding to the three parent nodes ` / `, ` / dir1`, and ` / dir1 / dir3`. Because the IX lock is applied to the MGL lock objects corresponding to the parent nodes, more compatible operations can be processed simultaneously, thus improving the concurrency of the file system.

[0128] Figure 6 This is a schematic diagram of a lock for changing operations provided by one or more possible embodiments of the present invention. Figure 6 This demonstrates how to apply a lock to an MGL lock object when an operation request instructs the creation of / dir1 / dir2. In this operation, a new node / dir1 / dir2 is created and treated as the target node. As shown in 630, the target node / dir1 / dir2 is mapped to the MGL lock object Lock. N As shown in 620, the parent node / dir1 maps to the MGL lock object Lock. N–2 As shown in 610, the root node / maps to the MGL lock object Lock1. Since this operation is a creation operation, it modifies the local state of the target node, therefore the MGL lock object Lock corresponding to the target node / dir1 / dir2 is also mapped. N Apply an X lock, and apply the MGL lock objects Lock1 and Lock2 corresponding to the two parent nodes / and / dir1. N–2 Apply IX locks separately. In this case, since / and / dir1 are locked in IX mode, other directories such as / dir1 / dir3 under the directory / dir1 can be accessed. Figure 6 (not shown in the image) or other directories such as / dir4, / dir5, etc. under the root directory ( Figure 6(Not shown in the diagram) This allows for simultaneous read operations (e.g., reading a directory or file) and modify operations (e.g., creating a file or directory, deleting a file). Therefore, more compatible operations can be processed concurrently, thus improving the file system's concurrency.

[0129] In one possible implementation, a lock may have already been applied to the first MGL lock object among multiple MGL lock objects, and the lock mode of the lock to be applied to the first MGL lock object may conflict with the lock mode of the lock already applied to the first MGL lock object. In this case, locks are not applied to the multiple MGL lock objects according to the operation request until the lock applied to the first MGL is released.

[0130] When a lock to be applied to an MGL lock object conflicts with a lock already applied to an MGL lock object, the operation request needs to wait for one or more other operations to release the lock already applied to the MGL lock object. Before the lock already applied to the MGL lock object is released, the operation request will not apply locks to multiple MGL lock objects, thereby avoiding interference with one or more other operation requests in the process.

[0131] MGL operations should follow the compatibility matrix shown in Table 1, where “√” indicates that the two lock modes are compatible and “×” indicates that the two lock modes conflict with each other.

[0132] Table 1

[0133]

[0134] Figure 7 This is a schematic diagram illustrating an example of concurrent operation provided by one or more possible embodiments of the present invention. Figure 7 An example of an MGL locking pattern for concurrent operations within the directory / dir1 is shown. Client 1's first operation request indicates an operation to read the contents of / dir1 / dir2, and client 2's second operation request indicates an operation to create / dir1 / dir3. In this case, the following locks are applied: based on client 1's first operation request, an IS lock is applied to / and / dir1, and an S lock is applied to / dir1 / dir2; based on client 2's second operation request, an IX lock is applied to / and / dir1, and an X lock is applied to / dir1 / dir3. Because IS and IX are mutually compatible, both operation requests can be processed simultaneously.

[0135] In one possible implementation, the operation request can indicate an operation affecting two or more target nodes. The lock mode of the lock to be applied to each of the multiple MGL lock objects can be determined based on one or more lock modes required by the respective MGL lock object. In this case, the multiple MGL lock objects may include one or more second MGL lock objects that each require one lock mode and one or more third MGL lock objects that each require two or more lock modes. The lock mode of the lock to be applied to each of the one or more second MGL lock objects can be determined as the lock mode required by the respective second MGL lock object, and the lock mode of the lock to be applied to each of the one or more third MGL lock objects is determined based on the most stringent lock mode among the two or more lock modes required by the respective third MGL lock object. In this way, complex operations affecting two or more target nodes can also be implemented using MGL, and complex operations can be executed concurrently with other compatible operations, thereby further improving the concurrency of the file system.

[0136] Table 2

[0137]

[0138] When an operation request indicates an operation affecting two or more target nodes (such as a move operation, rename operation, snapshot operation, etc.), a single node may require more than one lock mode. Among multiple MGL lock objects, there may be one or more MGL lock objects that each require only one lock mode, or one or more other MGL lock objects that each require two or more lock modes. For an MGL lock object requiring only one lock mode, the required lock mode can be considered as the lock mode to be applied to the MGL lock object. For an MGL lock object requiring two or more lock modes, the lock mode to be applied to the MGL lock object can be determined by summing all the lock modes required by the MGL lock object. The summing or aggregating of lock modes follows a lock mode algorithm, as shown in Table 2.

[0139] In one possible implementation, for an operation request affecting two or more target nodes, the lock mode to be applied to the corresponding MGL lock object can be determined as follows: Based on the operation request, obtain a first list of nodes requiring an X lock, a second list of nodes requiring an S lock, a third list of nodes requiring an IX lock, and a fourth list of nodes requiring an IS lock; map the four node lists to the corresponding MGL lock objects; superimpose one or more lock modes required by each of the multiple MGL lock objects, thereby obtaining a final lock mode for each of the multiple MGL lock objects. The first and second node lists are both target nodes. The third node list includes one or more parent nodes of the first node list, and the fourth node list includes one or more parent nodes of the second node list. These four node lists can share one or more common nodes. Depending on the specific complexity of the operation, there may be no nodes requiring S or IS modes; that is, the second and fourth lists can be empty.

[0140] To facilitate understanding of the technical solution of the locking method for locking a directory tree provided by this invention, some examples of complex operations will be described.

[0141] In one example, an operation request could instruct simultaneous read and write operations on HDFS snapshots within the same directory. For instance, to access / dir1 / dir2 / ... / dir M–1 / dir M Take a snapshot in the source directory / dir1 / dir2 / ... / dir M–1 / dir M Create a special destination directory within / dir1 / dir2 / ... / dir M–1 / dir M / .snapshot. In this case, / dir1 / dir2 / ... / dir M–1 / dir M and / dir1 / dir2 / ... / dir M–1 / dir M The ` / .snapshot` file represents the target node. The target nodes for reading content are ` / dir1 / dir2 / ... / dir`. M–1 / dir M An S lock is required for the path from / to / dir1 / dir2 / ... / dir M–1 The parent node requires an IS lock. The target node is / dir1 / dir2 / ... / dir M–1 / dir M The / .snapshot (i.e., the destination node of the operation) requires an X lock, while the path from / to / dir1 / dir2 / ... / dir M The parent node requires an IX lock. Then, the lock is applied as follows.

[0142] • Destination path / dir1 / dir2 / ... / dir M–1 / dir M X Lock of / .snapshot

[0143] • S lock + IX lock = path / dir1 / dir2 / ... / dir M–1 / dir M SIX lock

[0144] • IS lock + IX lock = from / to / dir1 / dir2 / ... / dir M–1 IX locks for all sub-paths.

[0145] Figure 8 This is a schematic diagram of a lock for snapshot operations provided by one or more possible embodiments of the present invention. Figure 8 This demonstrates how to apply a lock to an MGL lock object when an operation request instructs that an HDFS snapshot be taken on / dir1. This operation performs both read and write operations on / dir1 simultaneously. Figure 8 As shown in 810, the root node / maps to the MGL lock object Lock1. As shown in 820, the node / dir1 maps to the MGL lock object Lock. N–2 As shown in 830, the node / dir1 / .snapshot maps to the MGL lock object Lock. N–1 The node / dir1 is the target node for the read operation, therefore / dir1 requires an S lock, and its parent node / requires an IS lock. The node / dir1 / .snapshot is the target node for the write operation, therefore / dir1 / .snapshot requires an X lock, and its parent nodes / and / dir1 require IX locks. In this case, node / dir1 requires both an S lock and an IX lock, therefore, according to S + IX = SIX, a SIX lock must be applied to node / dir1. The root node / requires both an IS lock and an IX lock, therefore, according to IS + IX = IX, an IX lock must be applied to the root node / .

[0146] In another example, an operation request might instruct the file ` / dir1 / dir2 / readme.txt` to be moved to a different folder ` / dir1 / dir3`. For this operation, the node ` / dir1 / dir2 / readme.txt` is removed, and a new node ` / dir1 / dir3 / readme.txt` is created. Both ` / dir1 / dir2 / readme.txt` and ` / dir1 / dir3 / readme.txt` are affected by this operation and are therefore target nodes. Both target nodes require X locks because they have both been modified; one is removed, and the other is created. The target node ` / dir1 / dir2 / readme.txt` has three parent nodes: ` / `, ` / dir1`, and ` / dir1 / dir2`, which require IX locks. The target node ` / dir1 / dir3 / readme.txt` also has three parent nodes: ` / `, ` / dir1`, and ` / dir1 / dir3`, which require IX locks. For each parent node in ` / dir1 / dir2` and ` / dir1 / dir3`, since they are all parents of a target node, only one lock is needed, namely the IX lock. Therefore, the lock to be applied to the corresponding MGL lock object is the IX lock. For parent nodes ` / ` and ` / dir1`, they are the parents of two target nodes. As the parent of one target node, an IX lock is needed, and as the parent of the other target node, an IX lock is also needed. Therefore, since IX + IX = IX, the IX lock must be applied to the MGL lock objects corresponding to these two parent nodes ` / ` and ` / dir1`.

[0147] In another example, an operation request might instruct the file ` / dir1 / dir2 / readme.txt` to be renamed to ` / dir1 / dir2 / drink.txt`. This operation removes the node ` / dir1 / dir2 / readme.txt` and creates a new node ` / dir1 / dir2 / drink.txt`. Both ` / dir1 / dir2 / readme.txt` and ` / dir1 / dir2 / drink.txt` are affected by this operation and are therefore target nodes. Both target nodes require an X lock because they have both been modified; one is removed and the other is created. These two target nodes have three common parents: ` / `, ` / dir1`, and ` / dir1 / dir2`. In addition to both target nodes requiring an X lock, all parent nodes of these two target nodes require an IX lock according to IX + IX = IX.

[0148] In some cases, H0…H MSome values ​​may match, which could lead to lock contention within a single path. This contention can occur, especially when the array size N is relatively small. This lock contention, also known as a hashcode contention or mapping contention, refers to the situation where two or more nodes on a single path are mapped to the same MGL lock object due to identical hash code values. In other words, in the target node and one or more of its parent nodes, two or more nodes have the same hash code value and are therefore mapped to the same MGL lock object. In this case, the lock mode to be applied to the MGL lock object can be determined based on the lock modes required by the two or more nodes mapped to the MGL lock object: for example, by superimposing the lock modes required by the two or more nodes mapped to the MGL lock object. Superimposing lock modes also follows a lock mode algorithm, as shown in Table 2.

[0149] In one example, if nodes with the same hash code value correspond to locks with the same lock mode (e.g., IS or IX), the lock structure will remain the same as when there is no conflict. When nodes correspond to locks with different lock modes (e.g., IX and X, or IS and S), a more stringent level should be applied to that lock (X and S respectively), corresponding to the lock mode required by nodes at higher levels in the directory tree. In this case, another concurrent operation cannot apply the lock to that level until the incompatible lock is released.

[0150] When two or more nodes on a path are mapped to the same MGL lock object and the MGL lock object requires more than one lock mode, a more stringent lock mode can be applied to the MGL lock object to prevent the operation request from being interfered with by one or more other operation requests.

[0151] Figure 9 This is a schematic diagram illustrating an example of a conflict scenario provided by one or more possible embodiments of the present invention. Figure 9 Each circle in the diagram represents a node, and the number in parentheses inside the circle indicates the index of the MGL lock object corresponding to that node. Lock information is indicated in square brackets next to each node. The lock information includes a list of MGL lock objects used to lock the node in a specific lock mode. For each element in the list, one or more letters indicate the lock mode, and the number following one or more letters indicates the index of the MGL lock object.

[0152] For example, for the node / var / run, the circled "(4)" indicates that the node / var / run is mapped to MGL lock object 4. The lock information [x4, ix6, ix15] indicates that in order to lock / var / run in X mode, an X lock is applied to MGL lock object 4, and an IX lock is applied to MGL lock objects 6 and 15 (corresponding to their parent nodes / var and / , respectively). Similarly, the node / var / log corresponds to MGL lock object 1. In order to lock / var / log in X mode, an X lock is applied to MGL lock object 1, and an IX lock is applied to MGL lock objects 6 and 15 (corresponding to their parent nodes / var and / , respectively). Concurrent operations can apply an X lock [x4, ix6, ix15] to / var / run and [x1, ix6, ix15] to / var / log. On the paths / var / run and / var / log, each node is mapped to a different MGL lock object, so there is no conflict.

[0153] In addition, such as Figure 9 As shown, for the path / usr / bin, both the root node / and the node / usr / bin map to the same MGL lock object 15. The lock information [x15, ix17] for / usr / bin indicates that in order to lock / usr / bin in X mode, an X lock is applied to MGL lock object 15, and an IX lock is applied to MGL lock object 17 (corresponding to its parent node / usr). In this case, since both the root node / and the node / usr / bin map to MGL lock object 15, / usr / bin requires an X lock, and / requires an IX lock. Therefore, a stricter lock, namely an X lock, is applied to MGL lock object 15 (which can be obtained from Table 2 via IX+X=X). At this time, the root node / is also locked in X mode. Similarly, for the path / usr / lib, both / usr and / usr / lib map to MGL lock object 17. When / usr / lib requires an X lock while / usr requires an IX lock, a stricter lock, namely an X lock (obtainable from Table 2 via IX+X=X), is applied to MGL lock object 17. At this time, / usr is also locked in X mode. Note that applying an X lock to / usr / bin will not be processed simultaneously with applying an X lock to / var / run or / var / log, or applying an X lock to / usr / lib, because locking / usr / bin in X mode would cause the root node to also be locked in X mode, and an X lock on the root / is incompatible with any lock. The lock information [x15] next to the root node / is to indicate that when an X lock is applied to MGL lock object 15, the root node / will be locked in X mode, but it does not mean that the root node / can be locked in X mode when / var / run or / var / log is locked in X mode.

[0154] To facilitate understanding of the technical solution of the locking method for locking directory trees provided by this invention, some examples have been described. It should be noted that these examples are for illustrative purposes only and should not be construed as limiting the technical solution of this invention.

[0155] The following will describe embodiments of products related to the locking method for locking a directory tree.

[0156] Figure 10 A schematic diagram of a locking device provided by one or more exemplary embodiments of the present invention is shown. This locking device can be applied to a file system metadata management component that stores all metadata in memory, such as the NameNode in HDFS, or other components with similar functionality, and is not limited thereto. Figure 10 As shown, the locking device 1000 includes:

[0157] Module 1010 is used to obtain operation requests;

[0158] The processing module 1020 is used to perform MGL operations on multiple MGL lock objects in a multiple-granularity locking (MGL) lock object array according to an operation request. The MGL lock object array includes multiple MGL lock objects for recording the locking status of nodes in a directory tree. The multiple MGL lock objects correspond to multiple nodes in the directory tree. The multiple nodes include at least one target node of the operation request and one or more parent nodes of the at least one target node.

[0159] In one possible implementation, each node in the directory tree corresponds to a corresponding path in the file system, and each of the multiple nodes is mapped to a corresponding MGL lock object based on the information of the path corresponding to the corresponding node.

[0160] In one possible implementation, each of the multiple nodes is mapped to a corresponding MGL lock object based on the hash code value of the path corresponding to the respective node.

[0161] In one possible implementation, the hash code value is obtained by using a hash algorithm on at least one string associated with the path.

[0162] In one possible implementation, the MGL lock object array comprises N MGL lock objects, where N is an integer greater than 1.

[0163] In one possible implementation, each of the N MGL lock objects in the MGL lock object array has a unique index.

[0164] In one possible implementation, the indices of the N MGL lock objects are in the range [0, N–1].

[0165] In one possible implementation, each of the multiple nodes maps to the corresponding MGL lock object based on the remainder obtained by dividing the hash code value of the path corresponding to the corresponding node by N.

[0166] In one possible implementation, the processing module is specifically used to apply locks to multiple MGL lock objects according to an operation request, wherein the lock mode of the lock to be applied to the MGL lock object is determined according to the operation request.

[0167] In one possible implementation, if a lock has already been applied to the first MGL lock object among multiple MGL lock objects, and the lock mode of the lock to be applied to the first MGL lock object conflicts with the lock mode of the lock already applied to the first MGL lock object, then the locks on the multiple MGL lock objects are not applied according to the operation request until the lock already applied to the first MGL is released.

[0168] In one possible implementation, if the operation request indicates an operation affecting a target node, and the operation is a read operation, then an S lock is applied to the MGL lock object corresponding to the affected target node, and an IS lock is applied to each of the one or more MGL lock objects corresponding to one or more parent nodes of the affected target node.

[0169] In one possible implementation, if the operation request indicates an operation affecting a target node, and the operation is a write operation, then an X lock is applied to the MGL lock object corresponding to the affected target node, and an IX lock is applied to each of the one or more MGL lock objects corresponding to one or more parent nodes of the affected target node.

[0170] In one possible implementation, if the operation request indicates an operation affecting two or more target nodes, the multiple MGL lock objects include one or more second MGL lock objects that each require one lock mode and one or more third MGL lock objects that each require two or more lock modes. The lock mode of the lock to be applied to each of the one or more second MGL lock objects is determined to be the lock mode required by the corresponding second MGL lock object, and the lock mode of the lock to be applied to each of the one or more third MGL lock objects is determined based on the most stringent lock mode among the two or more lock modes required by the corresponding third MGL lock object.

[0171] In one possible implementation, if the target node and one or more of its parent nodes include two or more nodes mapped to the same MGL lock object, the lock mode of the lock to be applied to the same MGL lock object is determined based on the lock mode required by the two or more nodes.

[0172] One embodiment of the present invention provides a locking device. The locking device includes processing circuitry for performing the locking method described above for locking a directory tree.

[0173] One embodiment of the present invention provides a file system. The file system includes a combination of... Figure 10 The described locking device or a locking device including processing circuitry for performing the locking method described above for locking the directory tree.

[0174] One embodiment of the present invention provides a file system. The file system includes a metadata management component for performing the locking method described above for locking a directory tree.

[0175] One embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the computer-executable instructions cause the processor to perform the locking method described above for locking a directory tree.

[0176] One embodiment of the present invention provides a computer program product including computer-executable instructions. When the computer-executable instructions are executed by a processor, the computer-executable instructions cause the processor to perform the above-described locking method for locking a directory tree.

[0177] The embodiments described herein illustrate information sufficient for practice and demonstrate methods for practicing this subject matter. Upon reading the description with reference to the accompanying drawings, those skilled in the art will understand the concepts of the claimed subject matter and will recognize that the application of these concepts is not specifically mentioned herein. It should be understood that these concepts and applications are within the scope of this invention and the appended claims.

[0178] Furthermore, it is understood that any module, component, or device disclosing executable instructions herein may include or otherwise access one or more non-transitory computer / processor-readable storage media to store information, such as computer / processor-readable instructions, data structures, program modules, and / or other data. A non-exhaustive list of examples of non-transitory computer / processor-readable storage media includes magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, compact disc read-only memory (CD-ROM), digital video disc or digital versatile disc (i.e., DVD), Blu-ray Disc™ and other optical discs, or other optical storage devices; volatile and non-volatile, removable and non-removable media implemented in any method or technology; random-access memory (RAM); read-only memory (ROM); electrically erasable programmable read-only memory (EEPROM); flash memory or other storage technologies. Any such non-transitory computer / processor-readable storage medium may be part of a device or apparatus, or may access or be connected to a device or apparatus. Computer / processor-readable / executable instructions used to implement the methods, applications, or modules described herein may be stored by such non-transitory computer / processor-readable storage media or otherwise preserved.

[0179] In some aspects of the invention, a non-transitory computer-readable medium is provided that stores instructions which, when executed by a processor, enable the processor to implement the method of the invention.

[0180] In some aspects of the present invention, a device / chipset system is provided, the device / chipset system including at least one processor that executes instructions stored in a computer-readable medium to implement the method of the present invention.

[0181] This invention includes various embodiments, not only method embodiments but also other embodiments, such as apparatus embodiments and embodiments involving non-transitory computer-readable storage media. Embodiments may be combined individually or in combination with the features disclosed herein.

[0182] While this invention pertains to illustrative embodiments, it is not intended to be limiting. With reference to this document, those skilled in the art will clearly understand various modifications and combinations of the illustrative embodiments and other embodiments of the invention.

[0183] Features disclosed herein in the context of any particular embodiment may be implemented additionally or alternatively in other embodiments. For example, method embodiments may be implemented additionally or alternatively in apparatus, system, and / or computer program product embodiments. Additionally, while embodiments are described primarily in the context of methods and apparatus, other implementations are contemplated as instructions stored in one or more non-transitory computer-readable media, etc. These media may store programs or instructions to perform any of the various methods consistent with the present invention.

[0184] Furthermore, in this invention, the word "a" when used in conjunction with the word "comprising" in the claims and / or description may refer to "a," but also has the same meaning as "one or more," "at least one," or "one or more," unless otherwise expressly stated. Similarly, the word "another" may refer to at least a second or more, unless otherwise expressly stated. The word "and / or" as used herein, when used in conjunction with a list of items, refers to any one or more items included in that list.

Claims

1. A locking method for locking a directory tree in a file system, implemented by a locking device, characterized in that, include: Get the operation request; According to the operation request, an MGL operation is performed on multiple MGL lock objects in a multiple-granularity locking (MGL) lock object array, wherein the MGL lock object array includes multiple MGL lock objects for recording the locking status of nodes in the directory tree, the multiple MGL lock objects correspond to multiple nodes in the directory tree, and the multiple nodes include at least one target node of the operation request and one or more parent nodes of the at least one target node.

2. The locking method according to claim 1, characterized in that, Each node in the directory tree corresponds to a corresponding path in the file system, and each of the plurality of nodes is mapped to a corresponding MGL lock object based on the information of the path corresponding to the corresponding node.

3. The locking method according to claim 2, characterized in that, Each of the plurality of nodes is mapped to the corresponding MGL lock object based on the hash code value of the path corresponding to the corresponding node.

4. The locking method according to claim 3, characterized in that, The hash code value is obtained by using a hash algorithm on at least one string associated with the path.

5. The locking method according to claim 3 or 4, characterized in that, The MGL lock object array includes N MGL lock objects, where N is an integer greater than 1.

6. The locking method according to claim 5, characterized in that, Each of the N MGL lock objects in the MGL lock object array has a unique index.

7. The locking method according to any one of claims 1 to 6, characterized in that, The step of performing MGL operations on multiple MGL lock objects in the MGL lock object array according to the operation request includes: Locks are applied to the plurality of MGL lock objects according to the operation request, wherein the lock mode of the lock to be applied to the MGL lock object is determined according to the operation request.

8. The locking method according to claim 7, characterized in that, If a lock has already been applied to the first MGL lock object among the plurality of MGL lock objects, and the lock mode of the lock to be applied to the first MGL lock object conflicts with the lock mode of the lock already applied to the first MGL lock object, then the plurality of MGL lock objects shall not be locked according to the operation request until the lock already applied to the first MGL is released.

9. The locking method according to any one of claims 1 to 8, characterized in that, If the operation request indicates an operation affecting a target node, and the operation is a read operation, then a shared (S) lock is applied to the MGL lock object corresponding to the affected target node, and an intention shared (IS) lock is applied to each of the one or more MGL lock objects corresponding to one or more parent nodes of the affected target node.

10. The locking method according to any one of claims 1 to 9, characterized in that, If the operation request indicates an operation affecting a target node, and the operation is a write operation, then an exclusive (X) lock is applied to the MGL lock object corresponding to the affected target node, and an intention exclusive (IX) lock is applied to each of the one or more MGL lock objects corresponding to one or more parent nodes of the target node.

11. The locking method according to any one of claims 1 to 10, characterized in that, If the operation request indicates an operation affecting two or more target nodes, the plurality of MGL lock objects include one or more second MGL lock objects that each require one lock mode and one or more third MGL lock objects that each require two or more lock modes. The lock mode of the lock to be applied to each of the one or more second MGL lock objects is determined to be the lock mode required by the corresponding second MGL lock object, and the lock mode of the lock to be applied to each of the one or more third MGL lock objects is determined based on the most stringent lock mode among the two or more lock modes required by the corresponding third MGL lock object.

12. The locking method according to any one of claims 1 to 11, characterized in that, If the target node and one or more of its parent nodes include two or more nodes mapped to the same MGL lock object, the lock mode of the lock to be applied to the same MGL lock object is determined based on the lock mode required by the two or more nodes.

13. A locking device, characterized in that, include: The acquisition module is used to acquire operation requests; The processing module is configured to perform MGL operations on multiple MGL lock objects in a multiple-granularity locking (MGL) lock object array according to the operation request. The MGL lock object array includes multiple MGL lock objects for recording the locking status of nodes in the directory tree. The multiple MGL lock objects correspond to multiple nodes in the directory tree. The multiple nodes include at least one target node of the operation request and one or more parent nodes of the at least one target node.

14. The locking device according to claim 13, characterized in that, Each node in the directory tree corresponds to a corresponding path in the file system, and each of the plurality of nodes is mapped to a corresponding MGL lock object based on the information of the path corresponding to the corresponding node.

15. The locking device according to claim 14, characterized in that, Each of the plurality of nodes is mapped to the corresponding MGL lock object based on the hash code value of the path corresponding to the corresponding node.

16. The locking device according to claim 15, characterized in that, The hash code value is obtained by using a hash algorithm on at least one string associated with the path.

17. The locking device according to claim 15 or 16, characterized in that, The MGL lock object array includes N MGL lock objects, where N is an integer greater than 1.

18. The locking device according to claim 17, characterized in that, Each of the N MGL lock objects in the MGL lock object array has a unique index.

19. The locking device according to any one of claims 13 to 18, characterized in that, The processing module is used for: Locks are applied to the plurality of MGL lock objects according to the operation request, wherein the lock mode of the lock to be applied to the MGL lock object is determined according to the operation request.

20. The locking device according to claim 19, characterized in that, If a lock has already been applied to the first MGL lock object among the plurality of MGL lock objects, and the lock mode of the lock to be applied to the first MGL lock object conflicts with the lock mode of the lock already applied to the first MGL lock object, then the plurality of MGL lock objects shall not be locked according to the operation request until the lock already applied to the first MGL is released.

21. The locking device according to any one of claims 13 to 20, characterized in that, If the operation request indicates an operation affecting a target node, and the operation is a read operation, then a shared (S) lock is applied to the MGL lock object corresponding to the affected target node, and an intention shared (IS) lock is applied to each of the one or more MGL lock objects corresponding to the one or more parent nodes of the affected target node.

22. The locking device according to any one of claims 13 to 21, characterized in that, If the operation request indicates an operation affecting a target node, and the operation is a write operation, then an exclusive (X) lock is applied to the MGL lock object corresponding to the target node, and an intention exclusive (IX) lock is applied to each of the one or more MGL lock objects corresponding to the one or more parent nodes of the target node.

23. The locking device according to any one of claims 13 to 22, characterized in that, If the operation request indicates an operation affecting two or more target nodes, the plurality of MGL lock objects include one or more second MGL lock objects that each require one lock mode and one or more third MGL lock objects that each require two or more lock modes. The lock mode of the lock to be applied to each of the one or more second MGL lock objects is determined to be the lock mode required by the corresponding second MGL lock object, and the lock mode of the lock to be applied to each of the one or more third MGL lock objects is determined according to the most stringent lock mode among the two or more lock modes required by the corresponding third MGL lock object.

24. The locking device according to any one of claims 13 to 23, characterized in that, If the target node and one or more of its parent nodes include two or more nodes mapped to the same MGL lock object, the lock mode of the lock to be applied to the same MGL lock object is determined based on the lock mode required by the two or more nodes.

25. A locking device, characterized in that, Includes processing circuitry for performing the locking method for locking a directory tree according to any one of claims 1 to 12.

26. A file system, characterized in that, Includes a metadata management component for performing a locking method for locking a directory tree as claimed in any one of claims 1 to 12.

27. A computer-readable storage medium for storing computer-executable instructions, characterized in that, When the computer execution instructions are executed by the processor, the computer execution instructions cause the processor to perform the locking method for locking a directory tree according to any one of claims 1 to 12.

28. A computer program product comprising computer-executable instructions, characterized in that, When the computer execution instructions are executed by the processor, the computer execution instructions cause the processor to perform the locking method for locking a directory tree according to any one of claims 1 to 12.

Images