Managing lock coordinator rebalancing in a distributed file system

Abstract

Managing lock coordinator rebalancing in a distributed file system is provided herein. A node device of a cluster of node devices can include a processor and a memory storing executable instructions that, when executed by the processor, facilitate performance of operations. The operations can include determining an occurrence of a group change between the cluster of node devices and performing a probing function based on the occurrence of the group change. Further, the operations include reasserting a first lock of a group of locks based on a result of the probing function indicating a reassertion of the first lock. A second lock of the group of locks other than the first lock is not reasserted based on the result of the probing function. The cluster of node devices can operate as a distributed file system.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. non-provisional application No. 17 / 124,694, filed on December 17, 2020, entitled “Managing Lock Coordinator Rebalancing in a Distributed File System,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The subject matter is generally about distributed file systems. More specifically, the subject matter concerns managing lock coordinator rebalancing in a distributed file system, such as facilitating delayed lock assertions and cache invalidation during the lock coordinator rebalancing process. Background Technology

[0004] Distributed storage systems and / or object storage systems can provide a wide variety of storage devices while achieving high scalability, availability, and serviceability. The operation of a distributed file system (also known as a distributed storage system and / or distributed object storage system) may include locking mechanisms to maintain data consistency of associated shared files. When changes are made to nodes in the distributed file system (node ​​removal, node addition, first node removal and second node addition, etc.), lock coordination needs to be reallocated within the distributed file system. During this process, the distributed file system can be temporarily locked until the group change process is complete. Therefore, suspended operations are paused, which impacts the performance of the distributed file system and leads to a negative end-user experience.

[0005] The context described above regarding conventional storage systems is intended merely to provide an overview of the current art and is not intended to be exhaustive. The advantages of some embodiments, based on the following detailed review, further contextual description, and the various non-limiting examples described herein, may become more apparent. Summary of the Invention

[0006] The following is a simplified summary of the disclosed subject matter to provide a basic understanding of some aspects of the various embodiments. This overview is not a comprehensive overview of the various embodiments. It is neither intended to identify key or essential elements of the various embodiments, nor to depict the scope of the various embodiments. Its sole purpose is to present some concepts of this disclosure in a streamlined form as a prelude to the more detailed description that follows.

[0007] In this embodiment, the text provides a first node device within a cluster of node devices. The first node device may include a processor and a memory storing executable instructions that, when executed by the processor, facilitate the execution of operations. Operations may include determining that a group change has occurred among the node devices in the cluster and executing a probe function based on the occurrence of the group change. Furthermore, operations may include re-asserting the first lock of the lock group based on the result of the probe function instructing a re-assertion of the first lock. Second locks of the lock group, other than the first lock, are not re-asserted based on the result of the probe function. The cluster of node devices may operate as a distributed file system.

[0008] Based on the example, reasserting the first lock can include determining that the lock loss callback thread used the corresponding lock of the first lock and that the corresponding cache associated with the thread is available for invalidation.

[0009] In some implementations, the operation may include receiving a set of values ​​in response to a probe function before reasserting the first lock. A first value in the set of values ​​indicates that no cache is to be invalidated. A second value in the set of values ​​indicates that the lock loss callback thread does not utilize the lock and the corresponding cache associated with the thread is available for invalidation. A third value in the set of values ​​indicates that the lock loss callback thread utilizes the lock and the corresponding cache associated with the thread is available for invalidation. Further for these implementations, the operation may include determining that a second lock is associated with the first and second values, and determining that the first lock is associated with the third value.

[0010] According to some implementations, determining that a group change has occurred may include determining the removal of an existing node device from the cluster of node devices. Alternatively, determining that a group change has occurred may include determining the addition of a new node device to the cluster of node devices. In some cases, determining the addition of a new node device to the cluster of node devices may include both determining the removal of an existing node device and the addition of a new node device.

[0011] In the example, an incompatible lock is created when two threads hold exclusive locks on the same resource. In another example, executing the probe function could include executing a region function within a cluster that includes a coordinator node, bypassing node devices. According to yet another example, the operation could include discarding the block cache and inode cache associated with the second lock.

[0012] According to some implementations, the locks of a group of locks are associated with a lock coordinator node across a cluster of node devices. Further for these implementations, the operation includes enabling the tracking of defined lock types that are requested for each resource from the lock coordinator node.

[0013] Another embodiment of the method may include a first node device determining that a group change has occurred at the node device's group, comprising a portion of a group including processors and node devices. Furthermore, the method may include the first node device reasserting a first lock of the lock group based on the result of a probe function indicating a reassertion of a first lock. The probe function is executed based on the determination that a group change has occurred. The method may also include the first node device discarding a second lock of the lock group based on the result of a probe function indicating that a reassertion of a second lock is unnecessary. The first lock and the second lock are different locks in the lock group.

[0014] Depending on some implementations, reasserting the first lock may include determining that the lock was lost, the callback thread used the corresponding lock of the first lock, and that the corresponding cache associated with the thread is available for invalidation.

[0015] In some implementations, dropping the second lock may include determining that the lock loss callback thread does not use the corresponding lock of the first lock and that the corresponding cache associated with the thread is available for invalidation. Alternatively or additionally, dropping the second lock may include determining that there is no cache to be invalidated.

[0016] The method may include, before reassertion, executing a probe function that includes the execution of one or more zone functions to bypass the coordinator node included in the group of node devices. In the example, the group of locks is an exclusive lock in a distributed file system.

[0017] Group changes can include at least one of adding a node to a group on a node device and removing a node from a group on a node device.

[0018] Another embodiment relates to a non-transitory machine-readable medium, including executable instructions that, when executed by a processor, facilitate the execution of operations. The operations may include executing a probe function based on determining that group changes are occurring at a cluster of node devices configured for distributed file system operations. The probe function evaluates the associated cache of the group of corresponding locks and resources. The operations may also include determining, based on the result of the probe function, a first set of locks for which the corresponding locks should be re-asserted. Furthermore, the operations may include re-asserting the first set of locks, wherein a second set of locks for which the corresponding locks differs from the first set is not re-asserted based on the result of the probe function.

[0019] The reassertion of the first set of locks may include determining the corresponding lock in the first set of locks used by the thread that was lost, and that the corresponding cache associated with the thread is available for invalidation. Executing the probe function may include executing a region function when bypassing the coordinator node included in the cluster of node devices.

[0020] According to some implementations, before reasserting the first set of lock values, the operation includes receiving a set of values ​​in response to a probe function. A first value in the set of values ​​indicates that no cache is to be invalidated. A second value in the set of values ​​indicates that the lock loss callback thread does not utilize the lock and the corresponding cache associated with the thread is available for invalidation. A third value in the set of values ​​indicates that the lock loss callback thread utilizes the lock and the corresponding cache associated with the thread is available for invalidation. Furthermore, the operation may include determining that a second set of lock values ​​is associated with the first and second values, and determining that the first set of lock values ​​is associated with the third value.

[0021] To accomplish the foregoing and related objectives, the disclosed subject matter subsequently includes one or more of the features described more fully below. Certain illustrative aspects of this subject matter are set forth in detail in the following description and accompanying drawings. However, these aspects merely indicate some of the various ways in which the principles of the subject matter can be used. Other aspects, advantages, and novel features of the disclosed subject matter will become apparent from the following detailed description when considered in conjunction with the accompanying drawings. It should also be understood that this detailed description may include additional or alternative embodiments besides those described in this summary. Attached Figure Description

[0022] Various non-limiting embodiments are further described with reference to the accompanying drawings, wherein:

[0023] Figure 1 An example cluster of computing devices according to one or more embodiments described herein is illustrated;

[0024] Figure 2A The illustration shows an example non-limiting block diagram of resource allocation in an original cluster of a distributed file system prior to a group change, according to one or more embodiments described herein.

[0025] Figure 2B The diagram illustrates an example non-limiting block diagram of the resource allocation of a cluster in a distributed file system after nodes have been added, according to one or more embodiments described herein.

[0026] Figure 2C The diagram illustrates an example non-limiting block diagram of how resource allocation in a distributed file system changes after a node has been removed, according to one or more embodiments described herein.

[0027] Figure 3 The illustration shows a flowchart of an example non-limiting computer implementation of a method for re-asserting a delayed lock condition that should be satisfied after a group change, according to one or more embodiments described herein.

[0028] Figure 4The illustration shows a flowchart of an example non-limiting computer implementation of a method for managing lock coordinator rebalancing according to one or more embodiments described herein;

[0029] Figure 5 The diagram illustrates a flowchart of an example non-limiting computer implementation of a method for implementing delayed lock assertion and cache invalidation after a group change, according to one or more embodiments described herein.

[0030] Figure 6 The illustration shows a flowchart of an example non-limiting computer implementation of a method for selectively reasserting a lock after a group change, according to one or more embodiments described herein.

[0031] Figure 7 The illustration shows a flowchart of an example non-limiting computer implementation of a method for reasserting a first set of locks after a group change and a second set of locks without reasserting locks, according to one or more embodiments described herein.

[0032] Figure 8 The illustration shows an example non-limiting node device for managing lock resources in a distributed file system, according to one or more embodiments described herein.

[0033] Figure 9 The illustration shows an example non-limiting computing environment in which one or more embodiments described in the text can be facilitated.

[0034] Figure 10 The illustration depicts an example non-limiting network environment in which one or more embodiments described in the text may be facilitated.

[0035] Specific implementation methods

[0036] One or more embodiments will now be described more fully with reference to the accompanying drawings, which illustrate exemplary embodiments. In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the various embodiments. However, various embodiments may be practiced without these specific details. In other instances, well-known structures and devices are illustrated in block diagram form to facilitate the description of the various embodiments.

[0037] The example implementation points to lock assertions and cache invalidation that facilitate delays during lock coordinator rebalancing, such as based on group changes. As described herein, a probe function can be executed based on whether there is a cache to be invalidated, and if so, whether a lock loss callback thread needs the lock to invalidate the cache to return a value from a group. If the lock loss callback thread needs the lock to invalidate the cache, the lock can be re-asserted. Otherwise, the lock is not re-asserted.

[0038] An example computing platform that can optionally incorporate the lock reassertion technique disclosed in this paper is provided by DELL. ® ISILON OneFS provided by Inc. ® Clustering. It is understood that OneFS® Clustering is one of many optional cluster computing technologies, any of which can incorporate the teachings of this disclosure.

[0039] Figure 1 An example cluster of computing devices is illustrated, according to one or more embodiments described herein. Figure 1 Cluster 102, including data node devices, Figure 1 The term "M" refers to storage nodes 104(1), 104(2), ..., 104(M), where M is a positive integer. Each storage node 104(1), 104(2), ..., 104(M) may include a computing device. Storage nodes 104(1), 104(2), ..., 104(M) may be configured to serve objects in response to requests from client 108. Furthermore, typically one of the storage nodes 104(1), 104(2), ..., 104(M) may host a cluster controller virtual machine (not in the cluster controller virtual machine). Figure 1 As shown in the figure, this node becomes the cluster controller node that manages cluster 102. Storage nodes 104(1), 104(2)...104(M) can be coupled to each other via suitable data communication links including interfaces and protocols (e.g., but not limited to, Ethernet block 106).

[0040] Client 108 can send data system-related requests to cluster 102, which can typically be configured as a large object namespace. Cluster 102 can maintain an unlimited number of objects (e.g., up to trillions or more). For this end, nodes such as storage node 104 (2) typically include port 112, through which client 108 connects to cluster 102. Example ports 112 are provided for requests via various protocols, including but not limited to SMB (Server Message Block), FTP (File Transfer Protocol), HTTP / HTTPS (Hypertext Transfer Protocol), and NFS (Network File System); furthermore, SSH (Secure Shell) allows, for example, management-related requests.

[0041] Each node, such as node 104 (2), may include an instance of operating system 114 (e.g., OneFS® or another operating system). Each node, such as node 104 (2), may also include CPU 122, RAM 124, and storage devices such as disk 126. RAM 124 and disk 126 may include, for example, volatile memory, non-volatile memory, hard disk drive, solid-state drive, or other types of storage devices. Furthermore, RAM 124 and disk 126 at multiple storage nodes 104 (1) to 104 (M), as well as other storage devices attached to cluster 102, may be used to collectively support logical disks that provide shared storage locations for cluster 102.

[0042] It is important to note that cluster deployments can be of any size. Depending on the specific needs of the organization, some clusters may include five or fewer nodes, while large clusters may include a much larger number of nodes. The techniques disclosed in this document can be incorporated into clusters of any size.

[0043] In a non-limiting example, during a group change, the OneFS® Distributed File System experiences a tension between the initiator cache size and lock coordinator rebalancing. A large cache improves latency during file system operations. However, each entry in the initiator cache is protected by a lock from a two-layer locking system cached on the initiator node. During lock coordinator rebalancing, a new lock coordinator must be notified of every single one of these cached locks. This is called re-asserting the cached locks on the node. A large number of cached locks can cause unacceptable system hangs during a group change (e.g., the system is temporarily locked). The embodiments provided herein attempt to overcome the challenges documented above by delaying lock notification for almost all locks until after the group change has been completed. This allows for an increase in the initiator cache size while reducing group change time.

[0044] In more detail, OneFS® implements a two-layer distributed locking system (called "lk"). A two-layer distributed locking system means that locks on resources are coordinated by a given node (called the coordinator), which can request the lock from the initiator node on behalf of threads on the node (called the initiator). Therefore, when a thread requests a lock to perform work on a resource, the request may not only need to go to its local initiator, but also need to be sent from the node to the cluster-wide coordinator for that resource before the lock can be granted and the thread can be released.

[0045] Figure 2AThe illustration shows an example, non-limiting block diagram of resource allocation in an original cluster 200 of a distributed file system prior to a group change, according to one or more embodiments described herein. This example illustrates a four-node cluster (first node, second node, third node, and fourth node) before a group change. The size of each node represents the number of resources the node has. However, this is merely for illustrative purposes and the distributed file system is not limited to four nodes and the corresponding number of resources illustrated for each node. Furthermore, multiple resources can be evaluated for each node, which may depend on the node's storage capacity, the node's processing resources, the node's cache storage and / or system storage resources, etc. Therefore, some nodes may handle more responsibility for coordinating locks (e.g., coordinating more locks) compared to other nodes that handle fewer resources.

[0046] Figure 2B The diagram illustrates an example, non-limiting block diagram of the resource allocation of cluster 202 in a distributed file system after nodes have been added, according to one or more embodiments described herein. More specifically, Figure 2B The diagram illustrates the resource allocation after nodes have been added to the original cluster 200. In this case, a fifth node has been added. Although illustrated as adding a single node, in some cases more than one node can be added.

[0047] Figure 2C The diagram illustrates an example non-limiting block diagram of the resource allocation of cluster 204 in a distributed file system after a node has been removed, according to one or more embodiments described herein. More specifically, Figure 2C The illustration shows resource allocation after a node has been removed from the original cluster 200. In this case, the sixth node has been removed. Although illustrated as the removal of a single node, in some cases more than one node can be removed. Furthermore, depending on some implementations, one or more nodes can be added and / or one or more nodes can be removed from the original cluster (or a subsequent cluster). Resources previously allocated to the removed node can be reallocated to the remaining nodes.

[0048] Traditionally, various approaches to adapting to group changes have been considered. For example, consistent hashing has been applied to prevent any locks on the running coordinator from being moved while a node stops running. Implementations of consistent hashing can reduce the number of lock notifications that must be sent. In another example, during the notification process, any locks that are currently not protecting the state of other cached nodes while they are caching themselves can be discarded. Another approach to adapting to group changes is the drain service. Utilizing a drain service before a group change significantly reduces the size of cached data and the number of cached locks. However, this only works if the group change is expected and therefore does not help with unexpected socket disconnections between nodes.

[0049] The disclosed embodiments can delay lock notifications for almost all locks until the group change has been completed. Therefore, the disclosed embodiments can increase the initiator cache size while reducing group change time.

[0050] Figure 3 The illustration shows a flowchart of an example non-limiting computer implementation of a method 300 for a delayed lock re-assertion condition that should be satisfied after a group change, according to one or more embodiments described herein.

[0051] More specifically, at 302, the file system lock is re-asserted upon group change. Therefore, a re-assertion of the file system lock can be performed based on the certainty that a group change is scheduled to occur or is occurring. Such an assertion guarantees that, after a group change, no two threads in the system hold incompatible locks (e.g., two threads with exclusive locks on the same resource). This same guarantee can be provided at 304 by implementing a deferred lock re-assertion.

[0052] Delayed lock re-assertion can be achieved by satisfying various conditions. For example, as indicated in method 300 of the computer implementation at 306, the first condition could be that no thread will reacquire a lock cached before the group change until the lock has been re-asserted. As indicated at 308, the second condition could be that no cache protected by a cached lock will be evaluated until the lock has been reacquired. The third condition, as indicated at 310, could be that if another node holds an incompatible lock between the time of the group change and the time the lock is re-asserted, the cache will be discarded before any thread acquires that lock.

[0053] The lock coordinator can track the strongest lock for each resource holding within the window between group changes and reassertions, so that the lock coordinator can notify the initiator node that it needs to discard its cache.

[0054] Traditionally, when the lock coordinator requests the lock initiator to relinquish its lock, the initiator invalidates any caches protected by that lock in two steps. First, the initiator calls a probe function indicating whether any cache invalidation is required. If any cache must be invalidated, the lock system then calls a second function (lock loss callback, llcb) to invalidate that cache. The key difference between these two functions is that during the probe function, the thread running the probe does not hold the lock to be released. During the llcb, however, the thread holds the lock. Since the cache must be destroyed before any thread can acquire the lock, this is unacceptable when invalidating a cache during a failed lock assertion.

[0055] To address this issue, according to the various embodiments provided in this document, the probe function is modified to return three possible values ​​instead of two. The first value indicates no cache exists and invalidation is required. The second value indicates cache exists and invalidation is required, but the llcb thread does not request a lock. The third value indicates cache exists and invalidation is required, and the llcb thread requests a lock.

[0056] The return of a third value should be rare. Most block caches and inode caches can be discarded without retaining the locks protecting them. Exceptions are cases where data must be flushed to disk during the lock discarding process (e.g., OneFS® coalescer), or cases where the cache is protected by multiple locks in a non-trivial manner (e.g., OneFS® reverse cursor block).

[0057] By implementing the embodiment with the improved detection function provided in this paper, it is possible to achieve the following: Figure 4 The same modification discussed applies to the lock re-assertion during group changes. Figure 4 The illustration shows a flowchart of an example non-limiting computer implementation of a method 400 for rebalancing a management lock coordinator according to one or more embodiments described herein.

[0058] At 402 of the computer-implemented method, the system, including the processor, can run (or execute) all probe functions during group changes. These region functions should be faster than sending a message to the coordinator for each lock to re-assert the lock. Furthermore, at 404, the system can re-assert all locks for which probe() returned an indication that a cache exists to be invalidated and that the llcb thread requests the value of the lock (e.g., the third value discussed above).

[0059] At 406 of method 400 implemented in the computer, the system can enable the strongest lock type (shared, exclusive, etc.) to be tracked, with the strongest lock type being requested for each resource from the lock coordinator node. At 408, the system can complete the group change.

[0060] Figure 5 The illustration shows a flowchart of an example non-limiting computer implementation of a method 500 for implementing delayed lock assertions and cache invalidation after a group change, according to one or more embodiments described herein.

[0061] Following a group change, at 502, the system (e.g., via the initiator node) will continue to reassert locks while file system operations continue. At 504, any locks requested by the active file system thread will be prioritized for reassertions.

[0062] If a reassertion fails, at position 506 the system (e.g., via the lock initiator) can use the value previously returned by the probe function to determine whether to call llcb. The system will then discard the cached lock at position 508. Once the lock has been discarded, it can be acquired normally by a standard lock request. Once all nodes have completed assertions for all cached locks, the coordinator will be notified to discard their lock type traces.

[0063] Figure 6 A flowchart illustrating an example non-limiting computer implementation of a method 600 for selectively reasserting a lock after a group change, according to one or more embodiments described herein.

[0064] When a first node device in a group comprising processors and node devices determines that a group change has occurred in the node device's group, the computer-implemented method 600 begins at 602. The determination can be made before, during, or after the group change. A group change may include at least one of additional nodes added to the node device's group and nodes removed from the node device's group.

[0065] At position 604, the first node device can re-assert the first lock of the lock group based on the result of a probe function that instructs the first lock to be re-asserted. The lock group can be an exclusive lock in a distributed file system. The probe function can be executed by the first node device when it is determined that a group change has occurred at position 602 or later. Executing the probe function can include bypassing the coordinator node included in the group of the node device.

[0066] Depending on some implementations, reasserting the first lock may include determining that the lock was lost, the callback thread used the corresponding lock of the first lock, and that the corresponding cache associated with the thread is available for invalidation.

[0067] A second lock in a lock group can be discarded by the first node device. For example, a second lock in a lock group can be discarded based on the result of a probe function that indicates a reassertion that does not require a second lock. The first lock and the second lock are different locks in the lock group. In some implementations, discarding the second lock may include determining that the lock loss callback thread does not use the corresponding lock of the first lock and that the corresponding cache associated with the thread is available for invalidation. Alternatively or additionally, discarding the second lock may include determining that no cache is to be invalidated.

[0068] Figure 7 The illustration shows a flowchart of an example non-limiting computer implementation of a method 700 that reasserts a first set of locks after a group change and does not reassert a second set of locks, according to one or more embodiments described herein.

[0069] When a cluster of node devices configured to operate as a distributed file system detects a group change occurring between the node devices in the cluster, the computer-implemented method 700 begins at 702. Upon or after determining that a group change has occurred, at 704, the first node device executes a probe function. Executing the probe function may include executing a region function while bypassing the coordinator node included in the cluster of node devices.

[0070] At 706, the values ​​from the group of values ​​are received as the result of the probe function. The probe function can return one of three values. For example, the first value can provide initial information indicating that no cache exists to be invalidated. The second value can provide second information indicating that a cache exists to be invalidated, but the llcb thread does not request the lock. The third value can provide third information indicating that a cache exists to be invalidated, and the llcb thread requests the lock. It should be noted that although the terms first value, second value, third value, first information, second information, and third information, etc., are used in this document to distinguish values ​​and / or information from each other, they do not imply a particular order of indicating values ​​and / or information.

[0071] As indicated at 708, based on a value that is a first value or a second value, one or more locks associated with the first value or the second value are not re-asserted. Furthermore, at 710, based on a value that is a third value, the first node device re-assertes one or more locks associated with the third value. A second set of locks for locks that differ from the first set of locks is not re-asserted based on the result of the probe function.

[0072] Figure 8 The illustration shows an example non-limiting node device 800 for managing lock resources in a distributed file system, according to one or more embodiments described herein. The node device 800 may include... Figure 1One or more components and / or functions of a computing device and / or a computer-implemented method, and vice versa.

[0073] The devices (e.g., node device 800, etc.), systems, apparatuses, and / or processes explained in this disclosure may constitute multiple machine-executable components embodied within one or more machines (e.g., embodied in one or more computer-readable media (or media) associated with one or more machines). Such components, when executed by one or more machines (e.g., multiple computers, multiple computing devices, multiple virtual machines, etc.), can cause the multiple machines to perform the described operations.

[0074] In various embodiments, node device 800 can be any type of component, machine, device, facility, apparatus, and / or instrument that includes a processor and / or is capable of effectively and / or operatively communicating with wired and / or wireless networks. Components, machines, devices, facilities, apparatus, and / or instruments that may include node device 800 may include tablet computing devices, handheld devices, server-type computing machines and / or databases, laptop computers, notebook computers, desktop computers, cellular phones, smartphones, consumer appliances and / or instruments, industrial and / or commercial equipment, handheld devices, digital assistants, multimedia internet-enabled phones, multimedia players, etc.

[0075] Node device 800 may include an analysis component 802, a probe component 804, a lock coordinator manager component 806, a transmitter / receiver component 808, at least one memory 810, at least one processor 812, and at least one data repository 814. Analysis component 802 can determine whether a group change has occurred (or is occurring) between two or more nodes in a distributed file system. For example, analysis component 802 can determine the removal of one or more existing node devices from the cluster of node devices and / or the addition of one or more new node devices to the cluster of node devices.

[0076] The probe component 804 can be configured to execute (or have its execution performed by other components) a probe function. The probe function can be a region function that is faster than the traditional method of sending a message to the coordinator for each lock to re-assert the lock. The probe function indicates whether any cache invalidation is required. If some cache needs to be invalidated, a second function (called a lock loss callback or llcb) is called to invalidate that cache. During the probe function, the thread does not hold the lock to be released but runs the probe. During the llcb, the thread holds the lock. Holding the lock during the llcb is unacceptable when invalidating a cache during a failed lock assertion, because the cache should be destroyed before any thread acquires the lock.

[0077] Therefore, as discussed in this paper, the probe function executed by probe component 804 can return one of three possible values ​​(compared to only two values ​​associated with traditional probe functions). The first value can indicate that no cache needs to be invalidated. The second value can indicate that a cache needs to be invalidated and the llcb thread does not need a lock. The third value can indicate that a cache needs to be invalidated and the llcb thread needs a lock. Returning the third value can be rare because most block caches and inode caches can be discarded without maintaining locks protecting them. Exceptions include situations where data must be flushed to disk during lock discarding or where the cache is protected by multiple locks in a non-trivial manner.

[0078] When or after a third value is returned (e.g., a cache needs to be invalidated and the llcb thread needs the lock), the lock coordinator manager component 806 can reassert locks during (or after) a group change. To selectively reassert locks, the lock coordinator manager component 806 can reassert all locks for which the probe function returns a third value (a lock requested on the llcb thread). Furthermore, the lock coordinator manager component 806 can enable tracking of the strongest lock type (shared, exclusive, etc.) requested for each resource from the lock coordinator. Afterwards, the group change is complete.

[0079] After a group change, the initiator node can continue reasserting locks while file system operations continue. Any locks requested by the active file system thread will be prioritized for reassertion. If a reassertion fails, the lock initiator can use the value previously returned by probe() (e.g., a probe function) to determine whether to call llcb, and will then discard the cached lock. Once the lock has been discarded, it can be acquired normally by standard lock requests. Once all nodes have completed assertions of all cached locks, the coordinator will be notified to discard their lock type traces.

[0080] At least one memory 810 is operatively connected to at least one processor 812. The at least one memory 810 may store executable instructions and / or computer-executable components (e.g., analysis component 802, probe component 804, lock coordinator manager component 806, transmitter / receiver component 808, etc.), which, when executed by the at least one processor 812, can facilitate the execution of operations (e.g., operations discussed regarding various methods and / or systems discussed herein). Furthermore, the at least one processor 812 may be utilized to execute the computer-executable components (e.g., analysis component 802, probe component 804, lock coordinator manager component 806, transmitter / receiver component 808, etc.) stored in the at least one memory 810.

[0081] For example, at least one memory 810 may store the protocol associated with the lock for selective reassertion after a group change in the distributed file system discussed herein. Furthermore, at least one memory 810 may facilitate actions to control communication between node device 800 and other node devices, one or more file storage systems, or one or more devices, so that node device 800 uses the stored protocol and / or algorithm to achieve the improved overall performance of the distributed file system described herein.

[0082] It should be understood that the data storage (e.g., memory) components described herein can be volatile or non-volatile memory, or may include both. As a non-limiting example, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) used as external cache memory. As a non-limiting example, RAM has various available forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The memory disclosed is intended to include, but is not limited to, these and other suitable types of memory.

[0083] At least one processor 812 may facilitate the corresponding analysis of information regarding the rebalancing of the management lock coordinator in the distributed file system, such as facilitating delayed lock assertions and cache invalidation during lock coordinator rebalancing in the distributed file system. At least one processor 812 may be a processor dedicated to analyzing and / or generating received information, a processor controlling one or more components of the node device 800, and / or a processor that both analyzes and generates received information and controls one or more components of the node device 800.

[0084] The transmitter / receiver assembly 808 can be configured to send and / or receive data from, for example, nodes, probe functions, and / or communication devices. Through the transmitter / receiver assembly 808, the node device 800 can send and receive data simultaneously, at different times, or a combination thereof.

[0085] Referring to the flowcharts provided herein will provide a better understanding of the methods that can be implemented based on the disclosed subject matter. While the methods are shown and described as a series of boxes for the purpose of simplification, it should be understood and appreciated that the disclosed aspects are not limited by the number or order of the boxes, as some boxes may appear in different orders and / or substantially simultaneously with other boxes depicted and described herein. Furthermore, implementing the disclosed methods does not require all of the illustrated boxes. It is to be understood that the functionality associated with the boxes can be implemented by software, hardware, combinations thereof, or any other suitable means (e.g., devices, systems, processes, components, etc.). Additionally, it should be understood that the disclosed methods can be stored on an article of art to facilitate the transport and transfer of these methods to various devices. Those skilled in the art will understand and appreciate that these methods can alternatively be represented as a series of interrelated states or events, such as in a state diagram.

[0086] As used herein, the terms “storage device,” “node device,” “first node device,” “storage cluster node,” “storage system,” etc. (e.g., node device) can include, for example, private or public cloud computing systems used for storing data, as well as systems used for storing data including and not including virtual infrastructure. The term “I / O request” (or simply “I / O”) can refer to a request to read and / or write data.

[0087] As used herein, the term "cloud" can refer to a cluster of nodes (e.g., a collection of web servers), such as a cluster of nodes within a distributed object storage system, which are communicatively and / or operationally coupled to each other and host a collection of applications used to serve user requests. Typically, cloud computing resources can communicate with user devices via most wired and / or wireless communication networks to provide access to cloud-based services that are not locally stored (e.g., on the user device). A typical cloud computing environment can include multiple layers aggregated together that interact with each other to provide resources to end users.

[0088] Furthermore, the term "storage device" can refer to any non-volatile memory (NVM) device, including hard disk drives (HDDs), flash memory devices (e.g., NAND flash memory devices), and next-generation NVM devices, any of which can be accessed locally and / or remotely (e.g., via a memory attached network (SAN)). In some embodiments, the term "storage device" can also refer to a storage array comprising one or more storage devices. In various embodiments, the term "object" refers to a collection of user data of any size that can be stored across one or more storage devices and accessed using I / O requests.

[0089] Furthermore, a storage cluster may include one or more storage devices. For example, a distributed storage system may include one or more clients communicating with the storage cluster via a network. The network may include various types of communication networks or combinations thereof, including but not limited to networks using protocols such as Ethernet, Internet Small Computer System Interface (iSCSI), Fibre Channel (FC), and / or wireless protocols. Clients may include user applications, application servers, data management tools, and / or testing systems.

[0090] As used in this document, “entity,” “client,” “user,” and / or “application” can refer to any system or person that can send I / O requests to the storage system. For example, an entity can be one or more computers, the Internet, one or more systems, one or more businesses, one or more computers, one or more computer programs, one or more machines, machinery, one or more actors, one or more users, one or more consumers, one or more humans, etc., and will be referred to as entities or multiple entities as the context suggests.

[0091] In order to provide context for various aspects of the publicly available topic, Figure 9 The following discussion aims to provide a brief general description of the suitable environment in which various aspects of the publicly disclosed subject matter can be realized.

[0092] refer to Figure 9An example environment 910 for implementing the aspects of the foregoing topics includes a computer 912. The computer 912 may include a processing unit 914, system memory 916, and a system bus 918. The system bus 918 couples system components, including but not limited to system memory 916, to the processing unit 914. The processing unit 914 may be any processor among any available processors. Multi-core microprocessors and other multiprocessor architectures may also be used as the processing unit 914.

[0093] The system bus 918 can be any bus from several types of bus architectures, including memory bus or memory controller, peripheral bus or external bus, and / or any kind of local bus using available bus architectures, including but not limited to 8-bit bus, Industry Standard Architecture (ISA), Micro Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronic Devices (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), Universal Serial Bus (USB), Advanced Graphics Port (AGP), PCMCIA, and Small Computer System Interface (SCSI).

[0094] System memory 916 includes volatile memory 920 and non-volatile memory 922. The Basic Input / Output System (BIOS), which includes basic routines for transferring information between elements within computer 912 during startup, is stored in non-volatile memory 922. By way of non-limiting illustration, non-volatile memory 922 may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory 920 may include random access memory (RAM) used as external cache memory. By way of non-limiting illustration, RAM can take many forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct Rambus RAM (DRRAM).

[0095] Computer 912 also includes removable / non-removable, volatile / non-volatile computer storage media. Figure 9The illustration shows, for example, disk storage 924. Disk storage 924 includes, but is not limited to, devices such as magnetic disk drives, floppy disk drives, magnetic tape drives, Jaz drives, Zip drives, LS-100 drives, flash memory cards, or Memory Sticks. Furthermore, disk storage 924 may include a standalone storage medium or a combination of storage media with other storage media, including but not limited to optical disc drives such as optical disc ROM devices (CD-ROM), CD recordable drives (CD-R drives), CD rewritable drives (CD-RW drives), or digital versatile disc ROM drives (DVD-ROM). To facilitate connection of disk storage 924 to system bus 918, a removable or non-removable interface, such as interface 926, is typically used.

[0096] It should be understood that, Figure 9 Software is described that acts as an intermediary between a user and the basic computer resources described in a suitable operating environment 910. This software includes an operating system 928. The operating system 928, which may be stored on disk storage 924, is used to control and allocate the resources of the computer 912. System application 930 utilizes the resource management provided by the operating system 928 through program modules 932 and program data 934 stored in system memory 916 or on disk storage 924. It should be understood that one or more embodiments of the subject matter disclosed may be implemented using various operating systems or combinations of operating systems.

[0097] Users input commands or information into computer 912 through input devices 936. Input devices 936 include, but are not limited to, pointing devices such as mice, trackballs, pens, touchpads, keyboards, microphones, joysticks, game controllers, satellite dish antennas, scanners, TV tuners, digital cameras, digital video cameras, and webcams. These and other input devices are connected to processing unit 914 via system bus 918 through interface ports 938. Interface ports 938 include, for example, serial ports, parallel ports, game ports, and Universal Serial Bus (USB). Output devices 940 use some of the ports of the same type as input devices 936. Thus, for example, a USB port can be used to provide input to computer 912 and output information from computer 912 to output device 940. Output adapter 942 is provided to indicate that some output devices 940 require special adapters, such as monitors, speakers, and printers. By way of non-limiting illustration, output adapter 942 includes a video and sound card that provides a means of connection between output device 940 and system bus 918. It should be noted that other devices and / or device systems provide both input and output capabilities, such as (multiple) remote computers 944.

[0098] Computer 912 can operate in a networked environment using logical connections to one or more remote computers 944, such as multiple remote computers 944. The multiple remote computers 944 can be personal computers, servers, routers, network PCs, workstations, microprocessor-based applications, peer-to-peer devices, or other public network nodes, and typically include many or all of the elements described relative to computer 912. For simplicity, only storage device 946 is shown with the multiple remote computers 944. The multiple remote computers 944 are logically connected to computer 912 via network interface 948 and then physically connected via communication connection 950. Network interface 948 includes communication networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet / IEEE 802.3, Token Ring / IEEE 802.5, etc. WAN technologies include, but are not limited to, point-to-point links such as Integrated Services Digital Network (ISDN) and its variants, circuit-switched networks, packet-switched networks, and Digital Subscriber Line (DSL).

[0099] (Multiple) communication connections 950 refer to the hardware / software used to connect network interface 948 to system bus 918. Although communication connection 950 is shown internal to computer 912 for clarity, it can also be external to computer 912. For illustrative purposes only, the hardware / software required to connect to network interface 948 includes internal and external technologies such as modems including conventional telephone-grade modems, cable modems and DSL modems, ISDN adapters and Ethernet cards.

[0100] Figure 10This is a schematic block diagram of a sample computing environment 1000 with which the disclosed subject matter can interact. The sample computing environment 1000 includes one or more clients 1002. The clients 1002 may be hardware and / or software (e.g., threads, processes, computing devices). The sample computing environment 1000 also includes one or more servers 1004. The servers 1004 may also be hardware and / or software (e.g., threads, processes, computing devices). Servers 1004 may accommodate threads to perform transformations by employing one or more embodiments as described herein. One possible communication between clients 1002 and servers 1004 may be in the form of data packets suitable for transfer between two or more computer processes. The sample computing environment 1000 includes a communication framework 1006 that can be used to facilitate communication between clients 1002 and servers 1004. The clients 1002 are operatively connected to one or more client data repositories 1008 that can be used to store information local to the clients 1002. Similarly, (multiple) servers 1004 are operatively connected to one or more server data repositories 1010 that can be used to store local information of servers 1004.

[0101] Throughout this specification, references to "an embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. Therefore, the appearance of the phrases "in an embodiment," "in an aspect," or "in an embodiment" in various places throughout this specification does not necessarily refer to the same embodiment. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0102] As used in this disclosure, in some embodiments, the terms "component," "system," "interface," "manager," etc., are intended to refer to or include a computer-related entity or an entity associated with an operating device having one or more specific functions, wherein the entity may be one of hardware, a combination of hardware and software, software, or software in execution, and / or firmware. As an example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable program, an executing thread, computer-executable instructions, a program, and / or a computer. By way of non-limiting illustration, both an application running on a server and the server itself can be components.

[0103] One or more components may reside within a process and / or a thread of execution, and components may reside on a single computer and / or be distributed across two or more computers. Furthermore, these components may be executed from various computer-readable media on which various data structures are stored. Components may communicate via local and / or remote processes, for example, based on signals having one or more data packets (e.g., data from a component interacting with a local system, another component in a distributed system, and / or interacting with other systems across a network such as the Internet via the signal). As another example, a component may be a device having specific functions provided by mechanical parts operated by electrical or electronic circuitry, operated by a software application or firmware application executed by one or more processors, wherein the processors may be internal or external to the device and may execute at least a portion of the software or firmware application. Yet another example, a component may be a device providing specific functions through an electronic component without mechanical parts, the electronic component including a processor therein to execute software or firmware that at least partially endows the electronic component with the functions. In one aspect, components may be emulated via virtual machines (e.g., within cloud computing systems). Although the various components have been described as separate components, it will be understood that, without departing from the example embodiments, multiple components can be implemented as a single component, or a single component can be implemented as multiple components.

[0104] The terms “example” or “illustrative” are used herein to mean used as an example or illustration. Any embodiment or design described herein as “example” or “illustrative” is not necessarily to be understood as preferred or advantageous over other embodiments or designs. Rather, the use of the terms “example” or “illustrative” is intended to present the concept in a specific form. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X adopts A or B” is intended to mean either of the naturally inclusive arrangements. That is, “X adopts A or B” is satisfied in any of the above examples if X adopts A; X adopts B; or X adopts both A and B. Additionally, the articles “a” and “an” used in this application and the appended claims should generally be understood to mean “one or more” unless otherwise indicated or clearly pointed to from the context in the singular form.

[0105] Inference can also refer to techniques used to construct higher-level events from a set of events and / or data. Such inference leads to the construction of new events or actions from a set of observed events and / or stored event data, whether the events are closely related in time, and whether the events and data come from one or more event and data sources. Various classification schemes and / or systems (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, and data fusion engines) can be used in combination to perform automatic and / or inferential actions related to a publicly known topic.

[0106] Furthermore, various embodiments can be implemented as methods, apparatus, or articles of art that use standard programming and / or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed subject matter. The term "article of art" as used herein is intended to cover a computer program accessible from any computer-readable device, machine-readable device, computer-readable carrier, computer-readable medium, machine-readable medium, or computer-readable (or machine-readable) storage / communication medium. For example, computer-readable storage media may include, but are not limited to, radon-accessible memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, solid-state drives (SSDs) or other solid-state storage technologies, magnetic storage devices such as hard disks; floppy disks; magnetic stripes (multiple magnetic stripes); optical discs (e.g., optical discs (CDs), digital video discs (DVDs), Blu-ray discs (BDs)); smart cards; flash memory devices (e.g., card, stick, key drives); and / or emulated storage devices and / or virtual devices of any of the aforementioned computer-readable media. Of course, those skilled in the art will recognize that many modifications can be made to this configuration without departing from the scope or spirit of the various embodiments.

[0107] In one or more example embodiments, neither should it be assumed that the disclosed embodiments and / or aspects are exclusive to other disclosed embodiments and / or aspects, nor should it be assumed that the devices and / or structures are exclusive to the elements depicted in the example embodiments or multiple embodiments, unless the context explicitly states otherwise. The scope of this disclosure is generally intended to include modifications to the described embodiments, and, where appropriate, additions from other described embodiments, where appropriate, interoperability between described embodiments, and, where appropriate, the addition or subtraction of components(s) from one or more embodiments, where appropriate, the aggregation of elements (or embodiments) into a single device to achieve overall functionality, or, where appropriate, the distribution of functionality of a single device across multiple devices. Furthermore, the incorporation, combination, or modification of means or elements (e.g., components) depicted herein, or modifications of devices, structures, or subsets thereof that are not explicitly depicted herein but are known in the art or made apparent to a person skilled in the art by the context disclosed herein, are also considered to be within the scope of this invention.

[0108] The above description of the illustrative embodiments of this subject matter, including the content described in the abstract, is not intended to be exhaustive or to limit the disclosed embodiments to the precise forms disclosed. Although specific embodiments and examples have been described herein for illustrative purposes, various modifications are possible within the scope of such embodiments and examples, as will be appreciated by those skilled in the art.

[0109] In this regard, while the subject matter has been described herein in conjunction with various embodiments and corresponding figures, it should be understood where applicable that other similar embodiments may be used, or modifications and additions may be made to the described embodiments to perform the same, similar, alternative, or alternative functions of the disclosed subject matter without departing from the invention. Therefore, the disclosed subject matter should not be limited to any single embodiment described herein, but should be interpreted in accordance with the breadth and scope of the appended claims.

Claims

1. A system comprising: processor; as well as A memory storing executable instructions that, when executed by the processor, facilitate the execution of operations, including: Determine when a group change occurs between clusters of node devices; The probe function is executed based on the occurrence of the group change; In response to the probe function, a set of values ​​is received, wherein the set of values ​​includes a first value, a second value, and a third value, wherein the first value of the set of values ​​indicates that no cache needs to be invalidated, wherein the second value of the set of values ​​indicates that the lock loss callback thread did not use the lock and the corresponding cache associated with the thread can be invalidated, and wherein the third value of the set of values ​​indicates that the lock loss callback thread used the lock and the corresponding cache associated with the thread can be invalidated; The second lock in the lock group is determined to be associated with both the first and second values; Determine that a first lock in the group of locks, other than the second lock, is associated with the third value; and The first lock is re-asserted based on the result of the probe function that indicates the re-assertion of the first lock, wherein the second lock is not re-asserted based on the result of the probe function.

2. The system of claim 1, wherein reasserting the first lock comprises: The thread that determines the lock loss callback uses the corresponding lock of the first lock and the corresponding cache associated with the thread is available for invalidation.

3. The system of claim 1, wherein determining the occurrence of the group change comprises: Determine the removal of existing node devices from the cluster of node devices; Determine the addition of the new node device to the cluster of the node devices; or The removal of the existing node device and the addition of the new node device are both determined.

4. The system of claim 1, wherein an incompatible lock is created when two threads hold an exclusive lock on the same resource.

5. The system of claim 1, wherein executing the probe function comprises: When bypassing the coordinator node included in the cluster of the node devices, a region function is executed.

6. The system according to claim 1, wherein the operation further comprises: Discard the block cache and inode cache associated with the second lock.

7. The system of claim 1, wherein the locks of the group of locks are associated with a lock coordinator node among the clusters of node devices, and wherein the operation further comprises: This enables the defined lock type to be tracked for each resource requested from the lock coordinator node.

8. The system according to claim 1, wherein the cluster of node devices operates as a distributed file system.

9. A method comprising: The first node device, which is part of a group including processors and node devices, determines that a group change has occurred at the group of said node devices; In response to a probe function executed based on the ascertainment, the first node device receives a set of values, wherein the set of values ​​includes a first value, a second value, and a third value; The first node device determines that a second lock in the set of locks is associated with the first value and the second value, and the first lock in the set of locks is associated with the third value, wherein the second lock and the first lock are different locks in the set of locks; Based on the result of the probe function indicating the reassertion of the first lock, the first lock of the group of locks is reasserted by the first node device; as well as Based on the result of the probe function indicating that the reassertion of the second lock is not needed, the first node device discards the second lock, wherein the first value of the set of values ​​indicates that no cache is to be invalidated, wherein the second value of the set of values ​​indicates that the lock loss callback thread did not use the lock and the corresponding cache associated with the thread can be invalidated, and wherein the third value of the set of values ​​indicates that the lock loss callback thread used the lock and the corresponding cache associated with the thread can be invalidated.

10. The method of claim 9, wherein reasserting the first lock of the group of locks comprises: The thread that determines the lock loss callback uses the corresponding lock of the first lock and the corresponding cache associated with the thread is available for invalidation.

11. The method of claim 9, wherein discarding the second lock comprises: It is determined that the lock loss callback thread does not use the corresponding lock of the first lock and the corresponding cache associated with the thread is available for invalidation.

12. The method of claim 9, wherein discarding the second lock comprises: It is determined that there is no cache to be invalidated.

13. The method of claim 9, further comprising: Prior to the reassertion, executing the probe function includes: bypassing the coordinator nodes included in the group of node devices based on the execution of one or more region functions.

14. The method of claim 9, wherein the group of locks is an exclusive lock in a distributed file system.

15. The method of claim 9, wherein the group change includes at least one of adding an additional node to a group of the node device and removing a node from a group of the node device.

16. A non-transitory machine-readable medium comprising executable instructions, which, when executed by a processor, facilitate the execution of operations, including: A probe function is executed based on the determination that a group change is occurring at a cluster of node devices configured to operate as a distributed file system. The probe function evaluates the associated cache of a group of corresponding locks and resources. The corresponding locks include a first set of locks and a second set of locks that are different from the first set of locks. In response to the probe function, a set of values ​​is received, wherein the set of values ​​includes a first value, a second value, and a third value, wherein the first value of the set of values ​​indicates that no cache needs to be invalidated, wherein the second value of the set of values ​​indicates that the lock loss callback thread did not use the lock and the corresponding cache associated with the thread can be invalidated, and wherein the third value of the set of values ​​indicates that the lock loss callback thread used the lock and the corresponding cache associated with the thread can be invalidated; The second set of locks is determined to be associated with the first value and the second value, and the first set of locks is associated with the third value; The first set of locks that determines the corresponding lock based on the result of the probe function shall be re-asserted; as well as The first set of locks is re-asserted, wherein the second set is not re-asserted based on the result of the probe function.

17. The non-transient machine-readable medium of claim 16, wherein the first set of locks re-asserted comprises: The thread that determines the lock loss callback uses the corresponding lock from the first set of locks and the corresponding cache associated with the thread is available for invalidation.

18. The non-transient machine-readable medium of claim 16, wherein performing the probe function comprises: Execute the region function when bypassing the coordinator node included in the cluster of the node device.

Images