Method and system for shared memory access api in distributed system

Abstract

The present disclosure relates to methods and systems for shared memory access APIs in distributed systems. A system allocates, in a distributed system comprising a plurality of nodes, a plurality of memory portions comprising shared remote memory content. The system registers the allocated portions with an operating system for access via RDMA. The system accesses, by a first node, the allocated portions to obtain a local copy. The system performs, via a library structured atomic application programming interface call, an atomic operation on one or more bits of the shared remote memory content by one or more of: updating the one or more bits based on a new value and an offset; fetching, from the shared remote memory content, a current value of the one or more bits prior to the update based on the offset; and performing an action on the shared remote memory content based on a comparison of the fetched current value to an expected value in the local copy.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of management. More particularly, the present disclosure relates to a method and system for a lock-free cluster-wide shared memory access API based on libfabric atomics in a distributed system. BRIEF DESCRIPTION OF DRAWINGS

[0002] Figure 1 A diagram showing an exemplary environment for facilitating libfabric atomics (LFA) based shared memory access according to an aspect of the application is shown.

[0003] Figure 2 A diagram showing an exemplary environment for facilitating LFA based shared memory access according to an aspect of the application (including a shared bitmap) is shown.

[0004] Figure 3 A diagram showing a content table, a counter array, and exemplary communications associated with accessing a file in a file system according to an aspect of the application is shown.

[0005] Figure 4 A diagram showing an exemplary environment for facilitating the use of shared ring buffer queues to communicate messages using LFA based shared memory access in a distributed system according to an aspect of the application is shown.

[0006] Figure 5A A flow diagram showing a method of facilitating LFA based shared memory access in a distributed system according to an aspect of the application is presented.

[0007] Figure 5B A flow diagram showing a method of facilitating LFA based shared memory access in a distributed system according to an aspect of the application is presented.

[0008] Figure 6A A flow diagram showing a method of facilitating the use of LFA based shared memory access to access a file in a file system in a distributed system according to an aspect of the application is presented.

[0009] Figure 6B A flow diagram showing a method of facilitating the use of LFA based shared memory access to communicate messages via shared ring buffer queues in a distributed system according to an aspect of the application is presented.

[0010] Figure 6C A flow diagram showing a method of facilitating the use of LFA based shared memory access to communicate messages via shared ring buffer queues in a distributed system according to an aspect of the application is presented.

[0011] Figure 7An exemplary computer system that facilitates LFA-based shared memory access in accordance with an aspect of the present application is shown.

[0012] Figure 8 An exemplary apparatus that facilitates LFA-based shared memory access in accordance with an aspect of the present application is shown.

[0013] In the drawings, like reference numerals refer to same elements throughout. DETAILED DESCRIPTION

[0014] The following description is presented to enable any person skilled in the art to make and use the aspects and examples, and is provided in the context of particular applications and its requirements. Various modifications to the disclosed aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects and applications without departing from the spirit and scope of the disclosure. Thus, the aspects described herein are not intended to be limited to the aspects shown, but are to be accorded the widest scope consistent with the principles and features disclosed herein.

[0015] In a distributed system or cluster with multiple components on multiple devices (e.g., a high performance cluster (HPC) application in a large scale cluster computing environment), a key task is to synchronize access to cluster-wide shared variables and data structures. Implementation of a shared data access application programming interface (API) should strike a balance between guaranteeing consistency for all nodes and introducing as little additional latency as possible when accessing the shared entity for read, write, or update operations at the same time.

[0016] Some current cluster synchronization methods involve a distributed lock manager (DLM) and use a message-based system, for example, between a client and a server. However, as the number of nodes (e.g., in an HPC) increases, the increasing latency (due to sending, receiving, processing, and responding to messages, including acquiring and releasing locks) can gradually become unacceptable. Scalability can also be an issue. Increased load on the listening server and increased system size can cause bottlenecks. Other methods can use a shared message queue, but the performance of such a system can be limited due to the complexity of the implementation and the unavoidable two-way traffic.

[0017] Aspects described herein provide a system that addresses these challenges by providing cluster-wide lock-free atomic operations using library-structured atomic (LFA) operations on a shared memory region. Given a distributed system or cluster with many nodes, any node can access shared data without acquiring a lock, simply by using its local copy of the shared object to predict the outcome of the desired update and using a compare-and-swap (CAS) library-structured operation to ensure the consistency of the desired update. The system can distribute the update target addresses across the remote memories of multiple nodes by an algorithm that can bring the conflict rate under control. This can further result in reduced latency of performing operations and contention for shared resources. The described system can be used to implement various types of shared objects, such as counters, bitmaps, structured data, and cluster-wide synchronization primitives like spinlocks, semaphores, and ring buffer queues.

[0018] Shared atomic data

[0019] In a distributed system, such as a cluster file system like a structured attached memory file system (FAMfs), an important task is to maintain certain key data structures in memory and share them among all nodes that are actively using the file system. One solution is to keep all file system metadata in a shared consistent database. However, as the node count increases (as in HPC), the performance involved in using such a shared consistent database can drop exponentially. In aspects described herein, the metadata is still kept in a database and also loaded into memory at startup and subsequently operated on via library-structured atomic (LFA) API calls. The data structures accessed by these LFA API calls can remain relevant only while the FAMfs is active. The data structures can be reconstructed from the database each time at startup. Further, the system can perform any and all relevant updates to the database in the background using the FAMfs software such that the updates do not affect access to the shared metadata in the data structures via the LFA API calls. Because the database is never in the path of the LFA operations, it does not introduce any additional latency.

[0020] In use of LFA calls, from a logical perspective, an entity can resemble a client-server model, but without the server in the traditional sense of a client-server model. In aspects described herein, a "server" or "server node" can simply declare or allocate some region or portion of its own memory ("binary large object (blob)") as a shared object (which can be of any structure). The server node can register the allocated portion with the operating system, which informs the operating system that the allocated region needs to be locked in memory. The server node can also create access keys for "clients" or "client nodes." The server's responsibilities end here. The remaining operations can be performed by the "clients," as described below with respect to Figure 1

[0021] The system can operate on the shared object of the allocated binary large objects as a single contiguous space made up of all the allocated binary large objects. This single contiguous space can be referred to as an "LFA region" or "shared remote memory content." The described system can include multiple LFA regions that are all independent of each other, i.e., LFA binary large objects allocated by different sets of server nodes in a distributed system. The size of each binary large object allocated by a server can be different, i.e., can differ from the size of other binary large objects in a given LFA region.

[0022] To access shared data in a given LFA region, a node must attach to the LFA region via an LFA API call, which creates the necessary library structure connections to all servers of the given LFA region. The LFA API call can operate using global offsets within the given LFA region. Using a specific offset, the LFA API can compute the desired location in remote memory, i.e., the library structure address within a server that maps its memory chunks to the global space and the local offset within the given LFA region. The system can implement server address resolution based on a fast binary search in an ordered array of registered LFA binary large objects.

[0023] Only library structure calls can guarantee the consistency of library structure atomic operations. Therefore, all content updates in an LFA region must be done via library structure calls. For example, a client can only access an LFA region via LFA API calls, even if the target object or memory location is located or resides in the memory of that same client.

[0024] ​The target object is a 32-bit word or 64-bit word shared object at a given global offset within the LFA region. The system can support various operations on 32-bit words and 64-bit words. Batch Get / Put operations can provide load / store from / to local buffers to / from remote memory, respectively. These operations do not guarantee consistency and should only be used when concurrency is not an issue, for example, when the system is populating the LFA region from a database during startup.

[0025] The system can support simple arithmetic operations (such as adding or subtracting a value from a remote location) as well as logical bitwise operations (such as logical AND, OR, and exclusive OR (XOR) operations). Additionally, the system can support compare-and-swap operations, in which the system can compare a fetched remote value with a local value and replace the remote value with a new value if the fetched remote value matches the local value. Arithmetic and bitwise operations can support "execute-take" variants of the operations. Furthermore, atomic operations or commands can be implemented in hardware.

[0026] Because each client maintains its own complete local copy of the shared remote memory content, each client can navigate its local copy in a normal programming fashion without locking any content. When performing updates to the shared remote memory content, the client can activate LFA API calls to ensure that the updates are valid and consistent (e.g., using "execute-take" operations). If a conflict occurs (i.e., when more than one node attempts to update the same word in the remote memory at the same time), it is the responsibility of the client to recover from the conflict. In practice, in a highly parallel distributed environment, the chance of a conflict occurring can be low, which underscores the concept that, in some cases, it can be simpler or more efficient to request forgiveness than to request permission.

[0027] Figure 1 A diagram illustrating an exemplary environment 100 for facilitating shared atomic data in accordance with an aspect of the present application is shown. Environment 100 can include a plurality of "server" nodes, such as server node_0 138, server node_1 148, and server node_N 158, as well as a plurality of client nodes, such as client node_0 110 and client node_M 120 ("client nodes 110 and 120").

[0028] Each server node can allocate a "large binary object" of its local memory and register the allocated large binary object with the operating system for access via remote direct memory access (RDMA). For example, server node_0 138 can allocate LFA large binary object_0 130, server node_1 148 can allocate LFA large binary object_1 140, and server node_N 158 can allocate LFA large binary object_N 150. Each large binary object can include a plurality of bits, a set of one or more bits or words. For example, LFA large binary object_0 130 can include at least: word W[0] 132; word W[1] 134; and word W[2] 136. Similarly, LFA large binary object_1 140 can include at least word W[i] 142 and LFA large binary object_N 150 can include at least word W[n] 152. Thus, the system can create an "LFA region" or "shared remote memory content" that includes these allocated LFA large binary objects (130, 140, and 150).

[0029] Each client can then attach to the LFA region to obtain its own local copy of the shared remote memory content. For example, each of client nodes 110 and 120 can attach to the LFA region composed or consisting of LFA large binary objects 130, 140, and 150 to obtain its own local copy of the shared remote memory content (e.g., local copy 111 for client node_1 110 and local copy 121 for client node_M 120). Client nodes 110 and 120 can use the lfa_get() call to bulk load the shared remote memory content in the LFA region into their respective local buffers and begin processing the obtained local copy in their buffers.

[0030] The local copy 111 can include at least: word W[0] 112; word W[1] 114; word W[i] 116; and word W[n] 118. The local copy 121 can include at least: word W[0] 122; word W[1] 124; word W[i] 126; and word W[n] 128. The words in each local copy can correspond to the words in each LFA binary large object 130, 140, and 150. For example: word W[0] 122 in the local copy 121 can correspond to word W[0] 132 of the LFA binary large object 130 (and can be accessed / updated in the LFA region via communication 160); word W[i] 126 in the local copy 121 can correspond to word W[i] 142 of the LFA binary large object 140 (and can be accessed / updated via communication 162); and word W[n] 128 in the local copy 121 can correspond to word W[n] 152 of the LFA binary large object 150 (and can be accessed / updated via communication 164).

[0031] When a client node wishes to update a particular word within the LFA region, the system can issue an LFA API call (e.g., a library structure operation) based on a new value for the particular word and a global offset of the particular word within the LFA region, e.g., by using the new value and the global offset as parameters to the LFA API call. The system can use a local offset within the remote memory of the target server to determine the target server associated with the LFA binary large object storing the particular word, and can then perform a library structure atomic operation on the particular word (e.g., by performing the "desired action" of replacing the current value with the new value).

[0032] Alternatively, if the LFA API call is for an "execute-take" operation, the LFA API call can fetch the current value of the particular word in the remote memory of the target server, and then perform the desired action on the particular word. To verify that no conflict has occurred, the system can compare the fetched current value in the remote memory with the expected value in the local copy (e.g., the "local current value"). If the fetched current value does not match the expected value, the client can take appropriate action, e.g., to resolve the conflict.

[0033] Shared bitmap

[0034] Bitmaps are well-known structures, widely used in file systems and other applications that track free / used memory or storage space. Bitmaps can be implemented as local memory arrays, where a single bit corresponds to a block of data. A bit value equal to "1" indicates that the block is in use. A bit value equal to "0" indicates that the block is free. Since modern CPUs can implement various bit manipulation commands in hardware, manipulation of bits can be very efficient. However, manipulating individual bits in a bitmap that is shared among multiple nodes in a distributed system, such as an HPC cluster, can present challenges.

[0035] In a Fabric Attached Memory File System (FAMfs), the system can implement a shared bitmap using LFA API calls that tracks usage of media "extents" in an "extent bitmap." The FAM address space can be organized into a series of chunks, and FAM modules can be allocated in "extents." An extent can be the smallest unit of FAM allocation of a single FAM module. The FAMfs can organize extents (allocated from different FAM modules) into "slabs" to facilitate implementation of a particular data protection scheme (which can be set at the file level). A slab can be used as a container for a "stripe" (e.g., as a container for allocating a stripe). A stripe is a collection of chunks of data that have the same protection level as the corresponding container slab. A "slab map" is a data structure that can describe the geometry and configuration of the layout, i.e., which FAM extents are placed together to form a corresponding slab to support a particular protection scheme. An "extent bitmap" is a data structure that tracks whether a given extent is in use or free (on a FAM module or at the Non-Volatile Memory Host Controller Interface Specification (NVMe) device level).

[0036] In a FAMfs, both the slab map and the extent bitmap can be shared data structures. In a FAMfs, the system can implement extent bitmaps as a global bitmap array using LFA-based cluster-wide data structures. These extent bitmaps can support spatial allocation across all FAM modules without the need for cluster-wide locks. This global bitmap array can be spread across all nodes that represent the user process of the FAMfs to allocate FAM extents.

[0037] Certain nodes in a FAMfs can activate LFA API calls on the extent bitmap data structure. For example, input / output (I / O) nodes in a FAMfs can include allocator modules that allocate their particular shared memory region (for a slab map or an extent bitmap). These I / O nodes can be referred to as "allocator nodes." Each allocator node can also maintain its own local copy of the full bitmap, as described below with respect to Figure 2As described. That is, each allocator node can be responsible for its own FAM pool portion, which can ensure locality of the algorithm that performs allocation. However, the product of allocation, a bitmap of allocated FAM segments indicating used / free status, can be a global entity shared among all participating nodes (e.g., I / O nodes or allocator nodes in FAMfs). This can guarantee consistency of allocation-related information among all participating nodes. In contrast, the cost of locking the entire graph for each allocation would be unacceptably high. Thus, the current solution provides an efficient way to facilitate a shared global bitmap array across multiple allocation nodes in a distributed file system (e.g., FAMfs).

[0038] In FAMfs, each FAM module uses one segment bitmap, and the segment bitmap is shared only between the allocator node and each node that needs to access the segment map of all FAM modules in the FAMfs pool. Thus, the respective (each) server creates and allocates an LFA blob for a segment or portion of the global segment map, which is the responsibility of the respective server. At the same time, the respective (each) server attaches to each LFA blob in the LFA region, including its own allocated LFA blob. As described herein, to ensure CPU cache level consistency, the system performs all LFA operations through LFA API calls, even when the respective server wishes to access data stored in its own allocated LFA blob. In FAMfs, the system determines the partitioning of the LFA segment map based on the LFA API caller (e.g., FAMfs dynamic space allocator).

[0039] Figure 2 A diagram illustrating an exemplary environment 200 for facilitating sharing atomic data, including sharing bitmaps, is shown in accordance with an aspect of the present application. Environment 200 can include a plurality of nodes: allocator node_0 210; allocator node_1 220; and allocator node_N 230. Each allocator node can allocate its own "blob" or "bitmap segment." All allocated blobs or bitmap segments can be virtually or logically placed together to form an LFA region. Further, each allocator node can obtain its own local copy of all bitmap segments contained in the LFA region.

[0040] For example: allocator node_0 210 can allocate bitmap segment_0 212, which can include a number of bits; allocator node_1 220 can allocate bitmap segment_1 222, which can include a number of bits; and allocator node_N 230 can allocate bitmap segment_N 232, which can include a number of bits. The bits in these allocated bitmap segments can be represented by the blank boxes. LFA region 240 can represent a virtual bitmap that includes all of the bitmap segments (e.g., 212, 222, and 232) allocated by the allocator nodes (e.g., 210, 220, and 230).

[0041] Allocator node_0 210 can obtain a local complete bitmap copy 214 by attaching to virtual bitmap 240 (e.g., as indicated by arrows 266 and 274), which is further indicated as being attached to each of the allocated bitmap segments (e.g., 212, 222, and 232) (e.g., as indicated by the following: arrows 252 and 254 for bitmap segment_0 212; arrows 256 and 260 for bitmap segment_1 222; and arrows 262 and 264 for bitmap segment_2 232, respectively). Similarly, allocator node_1 220 can obtain a local complete bitmap copy 224 by attaching to the allocated bitmap segments in a similar manner (e.g., as indicated by arrows 270 and 276). Allocator node_2 230 can also obtain a local complete bitmap copy 234 by attaching to the allocated bitmap segments (e.g., as indicated by arrows 272 and 278).

[0042] An LFA API call to a section bitmap can implement a global find-first-clear-bit-and-set method that traverses the global bitmap to search for the first clear bit starting from some initial position. This initial position can be determined by an algorithm that aims to reduce contention between requesters (i.e., trying to spread the bits out). Since the requesting process always has a local copy of the entire bitmap in its local buffer, the system can first efficiently search the local buffer via built-in central processing unit (CPU) instructions (e.g., by inverting the value, and then performing the ffs() operation in x86). The ideal outcome is that the bit at the initial position is clear. However, if the bit at the initial position is not clear, the ffs() instruction can return the next available position. The API can then perform an atomic OR-and-fetch library structure call that attempts to propagate the new value to the global shared memory.

[0043] If the bit in the appropriate global memory location is cleared, the operation completes successfully. If the bit in the appropriate global memory location is not cleared, indicating that the bit was set by another actor or entity, the LFA call can return an error code and the contents of the global memory as they were just prior to the attempt to set or propagate the bit. This provides a recent snapshot of the current values of the bitmap, allowing the system to make intelligent decisions based on that information. Via an algorithm, the system can often attempt to stay within a local region, for example, staying on the same node determined by the initial value computed through a hash function. This can reduce I / O traffic in the fabric, and can also help reduce contention. Only when it finds that all bits in its "local" LFA region have been set can the system move to another node's LFA region.

[0044] It should be noted that the "get" portion of the LFA API call can always be performed. Thus, even in the case where the first attempt to set the bit succeeds, the system can return the updated contents or current values of the global memory, including all bits that can have been set by other operations. From the perspective of each allocator node, the segment bitmap can represent a self-healing local copy of the global shared memory.

[0045] Cluster-wide spinlock and semaphore

[0046] While the LFA API call can provide lock-free access to shared data (as described above), it is often limited to operating on individual memory words. In some cases, the system can need to guarantee consistent transactional access to multiple fields at once, for example, when a certain data structure needs to be traversed and subsequently extended or updated while maintaining consistency of more than one data word. Because the LFA API call is generally consistent only within the boundaries of a word, the system must regulate access to the entire structure in a different manner.

[0047] Aspects described herein regulate access to entire data structures by implementing a global spinlock. Spinlocks can be used in the kernel of a modern operating system to provide synchronization between multiple threads. In the described aspects, the system can create a global, cluster-wide spinlock based on the LFA API call. When a calling process needs to acquire the spinlock, the process can issue an LFA compare-and-swap (CAS) call to one of the LFA regions that holds a lock for a different purpose in a predefined LFA region. The CAS call can set a remote location to a certain value, but only if the remote location contains an initial value that matches an expected value (e.g., the current remote value fetched matches the local current value). If the fetched value matches the expected value, the operation is successful. The calling process can acquire the spinlock and proceed as needed.

[0048] If the fetched value does not match the expected value, this indicates that another process has acquired the spinlock and is working on the data protected by the spinlock. The calling process then reissues the CAS call and continues to "spin" in a loop until it either acquires the spinlock or the timeout period expires. When the calling process (or any other process that acquires the spinlock) finishes updating the data in the critical section, the process can release the spinlock by issuing a simple LFA write call, which can reset the value of the remote spinlock to the unlocked state.

[0049] Similar to global, cluster-wide spinlocks, LFA API calls can implement cluster-wide counting semaphores, which facilitate access to countable resources. Instead of using CAS calls, the semaphore API can also use atomic add-and-fetch and decrement-and-fetch calls. When a new free object is detected in a pool protected by a semaphore, the system can atomically increment the value of the semaphore by one. If a client or other entity needs to acquire an object in a pool protected by a semaphore, the client can attempt to decrement the semaphore by one. If the value of the semaphore (after the decrement attempt) is greater than zero, the operation can continue. If the value of the semaphore is not greater than zero (after the decrement attempt), this indicates that the semaphore was zero at the time of the call, and the client can retry the attempt to acquire the object at a later time.

[0050] As described herein, the implementation of global spinlocks and global semaphores are both based on LFA regions, and both of these synchronization primitives can leverage the LFA infrastructure. For example, the system can save both of these synchronization primitives in a special LFA region separate from other shared data, or as part of a global structure that contains both the lock and the protected data structure, as described below with respect to Figure 3

[0051] Content table and counter array

[0052] In large systems, such as HPC applications and distributed systems or clusters, a key task is to track the usage of resources for statistical performance analysis and actual resource planning and distribution. This can be a challenging task because access to these resources in HPC applications can be massively concurrent. The above description relates to implementing atomic counters using LFA APIs, but in a system with thousands of nodes creating hundreds of files concurrently, allocating these counters can be a challenging task.

[0053] ​One aspect of the described system provides a simple and reliable mechanism to allocate various usage counters across the cluster. The system provides a table of contents (TOC) to keep track of all existing counters. The system also allocates memory blocks to contain the counters themselves. The system can partition the TOC and the counter blocks (or blobs) in the same way as regular LFA regions to spread the traffic of the structure across multiple server nodes. The system can use a global spinlock to protect each TOC partition.

[0054] In the example of FAMfs, each file has a unique identifier ("FileID"), as described below. According to the general principles of LFA, a client can perform a TOC lookup to search for a particular FileID in a particular LFA by first looking in its local copy of the TOC without acquiring the global spinlock. If the given FileID is found in the local copy of the TOC, then the file is likely already stored in the corresponding global memory location. The client can then acquire the spinlock on the TOC partition belonging to the given LFA and verify this. If the file has been processed by another process and the global TOC entry is empty or contains a different FileID, then the system can perform a full scan of the TOC and allocate a new TOC entry. If the given FileID is not found in the local copy of the TOC, then the system can acquire the global spinlock on the TOC partition belonging to the given LFA based on an algorithmic decision. The system can then search for the FileID in the TOC's records, e.g., for an empty entry or an entry with the given FileID if it has already been allocated by another client process. Similar to the general principles of the LFA design, all operations related to the counters are unilateral, so there is no message exchange between the client and the server. All operations are atomic RDMA transactions from the client to the server node's memory.

[0055] In FAMfs, the file system must keep track of open files and updates to these open files. FAMfs is a checkpoint-recovery oriented file system, so it must also keep track of when files are "layered," i.e., when the erasure code (EC) computation for a file is completed and the file is protected from system, device, or FAM module failure. However, because the EC encoding process is not performed on files but on physical data chunks in the FAM module, aspects described herein provide an alternative way to track the layering process. One solution is to simply allocate a predefined number of counters. However, this can be a waste if too many counters are allocated, or extremely inefficient if client processes need to wait for a counter to become available.

[0056] In FAMfs, each file has a unique identifier ("FileID") derived from the fully-qualified name of the file. This FileID can be used to track all counters associated with the file, e.g.: open count; write count; etc. When the system issues an open_file() call, the client process first searches for the FileID (e.g., "X") in the TOC for the specified LFA region. The TOC LFA can be shared among all server nodes (as described above with respect to Figure 1 Figure 3 Similar to the general LFA principle, the system aims to distribute the load as much as possible through an algorithm. The system can use the FileID as input to a hash function that can distribute the load evenly among all nodes serving a given LFA region. The hash function can provide a high probability that multiple open_file() calls from the same client node will be directed to different server nodes, which can result in keeping equal load on all I / O nodes in the cluster. Thus, the hash function can determine that a particular server is responsible for a particular file based on the unique FileID of the file. The system can convert the file name to its unique FileID, which points to a particular LFA region, similar to the segment bitmap described above with respect to Figure 2

[0057] Figure 3 ​​A diagram 300 showing a table of contents 320, a counter array 340, and exemplary communications associated with accessing a file in a file system, according to an aspect of the present application. Diagram 300 can include a requestor_l 310 (e.g., a client process), a table of contents (TOC) 320, and a counter array 340. Requestor_l 310 can perform a series of steps (311-315) to access a particular file. TOC 320 can include an associated spinlock (SL) 322 and entries indicating at least: a file ID 324 (e.g., FileID) for a given file; a reference count 326, which can indicate how many clients currently have the given file open; and an offset 328, which can be an offset in counter array 340. Counter array 340 can be an array of counters allocated for maintaining a given FileID. Entry 330 in TOC 320 can include a FileID of "0," a reference count of "0," and an offset of "0," which can indicate an empty entry. Entry 332 in TOC 320 can include a FileID of "X," a reference count of "RC X," and an offset of "OFF X." Similarly, entry 334 in TOC 320 can include a FileID of "Y," a reference count of "RC Y," and an offset of "OFF Y." The reference counts and offsets can be integer values.

[0058] When a client process wishes to have the system open a file with FileID = "X", the client process can first compute the TOC to which this FileID belongs, e.g., by using a hash function. As described above, the client can first perform a lookup in its local copy of the TOC to search for the FileID without acquiring the global spinlock. If the FileID does not exist in the local TOC, the client process can determine that the TOC 320 of a particular LFA (not shown) is the appropriate TOC. The client process can acquire the spinlock 322 of the TOC 320 (as indicated by the step_1 311 and acquire 350 communications). The client process can scan the TOC 320 for a FileID of "X" (as indicated by the step_2 312 and scan 352 communications). The scanning process can start at entry 330. If the client process does find a FileID of "X" (i.e., a matching entry 332), the system can increment the reference count (as indicated by the step_3 313 and increment reference count 354 communications). If the client process does not find a FileID of "X" (e.g., if entry 332 does not exist in the TOC 320), the system can create an entry for the FileID of "X", increment the reference count by one, initialize any other relevant counters, and record the selected offset in the counter array 340 in the new TOC entry (e.g., by using the first available or other manner of determining an element in the counter array 340 in which to track the current value of the counter itself).

[0059] Subsequently, the client process can release the spinlock 322 (as indicated by the step_4 314 and release 356 communications), which makes the TOC 320 available to any other requesting client process, node, or entity. At this point, the client process can now directly access and manipulate the counters in the counter array. For example, the client process can increment the appropriate counter in the counter array 340 (as indicated by the step_5 315 and increment counter 358 communications). The client process can determine the location of the appropriate counter in the counter array 340 by using the offset 328 of the matching entry 332 (as indicated by the dashed arrow 362 from entry 332 to the element of the counter array 340 indicated as "Counter[j]" 346).

[0060] In this manner, once a file is opened, the client process can use one or more counters to determine, for example, how many writes have occurred (first counter array) versus how many writes have been committed (second counter array). The system is not limited to one or two counter arrays, and can use more or less as desired. The system can wait for the parity thread to perform parity calculations on a certain number of blocks corresponding to the file. After completing parity calculations for all such blocks, the system can determine that the file is ready for lamination.

[0061] The system can access these counters frequently, as processing data blocks can result in accessing and incrementing the appropriate counter. Using LFA API calls and scattering these TOCs and counter arrays into separate LFA regions can minimize contention for the counters. The delay involved in performing steps 311-314 (in the OPEN file) need only occur once, when the file is opened. Once the client process has determined the offset, the client process can directly access the counters without any further locking, i.e., can use LFA API calls in a lock-free cluster-wide manner. This short delay in opening / closing a file does not impact performance in any significant way, as the system is performing many other operations in parallel with the TOC lookup, which can hide or cover this additional delay. Figure 3

[0062] When the client process wishes for the system to close an open file, the system again acquires the SL 322, locates the entry for the requested FileID, and decrements the reference count by one. If the reference count is "0", indicating that no other entity, process, or node in the cluster has the file open, the system can delete the corresponding offset in the entry as well as the associated element or block in the counter array 340. The system can perform the appropriate updates and subsequently release the spinlock 322.

[0063] Thus, for a given file, the client process need not know the actual physical location of the counters. Using the steps and operations described above, the client process can simply look up the relevant information in the determined TOC and proceed in the manner described above.

[0064] Global ring buffer queue

[0065] While the above aspects address atomic operations on certain remote shared memory content (by eliminating message passing), nodes in an HPC cluster still must perform message passing (e.g., by using message queues), as this is an essential task in any distributed application. Generally speaking, message queue implementations are relatively complex and require a significant amount of resources. The described aspects of the system use a simplified mechanism to provide a shared queue, so that short, fixed-size messages can be quickly transferred from node to node in a lower resource consumption manner.​

[0066] The system is based on unilateral communication as described above for LFA API calls and operations, and uses a ring buffer queue (RBQ). The RBQ is a distributed structure that is partitioned into "segments" across multiple servers, similar to a section bitmap or data contained in an LFA binary large object that constitutes an LFA region. Each RBQ segment can include: a block of memory holding messages being transmitted; two semaphores for synchronization of access; and two pointers / counters for tracking locations related to insertion and removal operations, as described below with respect to Figure 4

[0067] Thus, logically, a single shared RBQ can include multiple partitions or segments, each serviced by a separate server node, similar to the way each LFA binary large object is serviced by a separate server node. The details of this implementation are invisible to the clients. The clients can simply attach to the queue by name. The RBQ can include an API that can automatically handle routing of incoming requests.

[0068] Each server in the distributed system (e.g., each I / O or distributor node in FAMfs) can create a segment for a particular queue, and can also maintain an LFA region that holds all the necessary components for the created segment or that holds all the necessary components associated with the created segment. The system can fix the size of the RBQ at startup. At the time of creating a segment or a queue, the system can also initialize all the necessary components through each respective server node. The system can create multiple queues, and any node can be both a server for one RBQ and a client for another RBQ. A client can attach to an existing queue, which can cause a bump in the atomic usage counter on the server to track the overall usage of the queue and prevent unexpected behavior when exiting the queue.

[0069] Figure 4 An exemplary environment 400 for facilitating the transmission of messages using shared ring buffer queues is shown in accordance with an aspect of the present application. Environment 400 can include a plurality of clients (e.g., clients CO-C N ) and a plurality of servers, including: server_0 410; server_1 420; and server_K 430. Each server can create an RBQ segment, which can include, comprise, contain, or be a queue. For example: server_0 410 can create RBQ segment_0 412; server_1 420 can create RBQ segment_1 422; and server_K 430 can create RBQ segment_K 432.

[0070] ​Clients can attach to RBQ segments to obtain local copies of RBQs, similar to the way that clients can attach to each LFA binary large object or segment bitmap segment to obtain respective local copies (as described above with respect to Figure 1 and Figure 2 respectively). For example: client C3 can attach to RBQ segment_0 412; client C7 can attach to RBQ segment_1 422; and client C N K 432.

[0071] Figure 4 The lower half depicts a detailed view of the communication between client C N 450 and server_K 430. Client C N 450 can include an RBQ push module 452 that handles access to a particular RBQ segment, and server_K 430 can include an RBQ pop module 440 that handles access to a particular RBQ segment. RBQ push 452 and RBQ pop 440 can be modules, processes, components, or units implemented in middleware, software, or hardware.

[0072] Server_K 430 can also include RBQ segment_K 432, which can include: a memory block 433 (e.g., an array 433 or a queue 433) that can hold data / messages transmitted in elements such as W[0] 434, W[1] 436, and W[2] 438; an input semaphore (IS) 462 and an output semaphore (OS) 464 for synchronizing access to queue 433; and an input pointer (IP) 460 and an output pointer (OP) 466 for tracking the location for inserting and removing messages from queue 433.

[0073] Upon performing initialization and other start-up processes, the system can set IS 462 to the maximum size of queue 433. The system can set OS 464 to “0” to indicate that no data (e.g., messages) are stored in queue 433. The system can also set both IP 460 and OP 466 to “0” as initial values.

[0074] During operation, a client (e.g., client C N 450) that wishes to place a message in a ring buffer queue (e.g., queue 433) can first attempt to acquire IS 462. The value of IS 462 can indicate the number of free slots in the queue. If the value of IS 462 is greater than “0”, then client C N450 can acquire IS 462 by decrementing the value of IS 462 by "1" (as indicated by step 471 (IS— )). If the value of IS 462 is "0" (indicating that there are no free or available slots in RBQ or queue 433), then the client must wait until the server finishes processing the request and releases IS 462. The wait semaphore becomes available is described below.

[0075] Upon successful acquisition of IS 462, the client can increment IP 460 (as indicated by step 472 (IP++)). Rather than IP 460 pointing to W[l] 436 (as indicated by the dashed curved arrow 477), IP 460 now points to W[2] 438 (as indicated by the bold curved arrow 476). Since the RBQ is a circular buffer, the system can set IP 460 to "0" again when the end of the buffer (i.e., queue 433) is reached. Client C N 450 can send the data (as a message) to the memory slot pointed to by IP 460 via RDMA (as indicated by step 473). Recall that each message in RBQ segment_K 432 is of the same fixed size. Thus, the system can use a simple offset-based calculation to compute the location of this memory (corresponding to W[2] 438). Finally, client C N 450 can release OS 464 by incrementing the value of OS 464 by "1" (as indicated by step 474 (OS++)). This increment can notify server_K 430 that there is data ready to be processed in queue 433.

[0076] During normal operation, server_K 430 can wait on OS 464, which has a value of "0" when queue 433 is empty. When OS 464 is incremented by client C N 450, server_K 430 can acquire OS 464 by decrementing the value of OS 464 by "1" (as indicated by step 481 (OS— )). Server_K 430 can continue to fetch the message stored at the location pointed to by OP 466 (i.e., W[2] 438) via RDMA (as indicated by step 482). Upon fetching the message or data from queue 433, server_K 430 can increment OP 466 by "1" (as indicated by step 483 (OP++)), such that OP 466 no longer points to W[2] 438 (as indicated by the solid curved arrow 486), but now points to the next element in queue 433 (as indicated by the dashed curved arrow 487). Similar to IP 460 for client C N450 works, when the OP 466 reaches the end of the queue 433, the OP 466 can wrap around to the beginning of the circular RBQ segment. To avoid tying up valuable resources, the server_K 430 can copy the data from the global memory to its local buffer (not shown) and can subsequently free the IS 462 by incrementing the value of the IS 462 by "1" (as indicated by step 484 (IS++)), which indicates that a free slot has become available. It should be noted that while the control information for each queue (e.g., IP 460, OP 466, IS 462, and OS 464) can be stored in the LFA region, it is not necessary to store the data segment for each queue in the LFA region. The data segment can only exist on the "server" node, i.e., for the process that owns the particular queue segment. As noted above, when the IP / OP state is determined by an atomic transaction, the client can perform a regular RDMA read / write from / to the respective data segment defined by these pointers. Thus, the client does not need to keep a local copy of the entire queue, as the atomicity of the RDMA is guaranteed by the queue semaphore in the LFA.

[0077] Waiting for a semaphore to become available can be implemented in different ways. A first implementation for a process wishing to acquire a semaphore is to simply spin on an atomic CAS call until the value changes. The wait involved in this spin time is localized. That is, while the system can waste local CPU cycles while waiting, the spin time does not generate traffic in the fabric, as the value being checked resides in memory on the same node. However, this approach still consumes CPU cycles.

[0078] A second implementation is to use regular messaging between the client and the server, e.g., sending a notification or wake-up message to the server after the client has inserted an element into a queue that was previously empty. In this implementation, the server can go to sleep immediately after its queue is exhausted and wait for a wake-up message. This solution does introduce an initial delay upon waking up, but can reduce resource usage when the system is idle. In scenarios with a lot of messages, the server does not go to sleep, no wake-up messages are delivered, and the system can run at high speed. However, this implementation still involves an initial delay upon waking up. The system can offset this initial delay by sending a wake-up message to the server only when a certain or predetermined number of messages exist in the queue, rather than sending a wake-up message every time the queue size reaches "1" (or some other number less than the predetermined number).

[0079] A third implementation is to utilize a library structure RDMA counter to facilitate server wake-up. Some library structure providers allow the use of a "passive" RDMA counter, where if a certain memory region is accessed (e.g., written or read), the counter can change its value. This implementation can support the use of separate counters for read and write operations. Upon emptying the queue, the server's write counter can go to sleep. The client can place data or messages into the RBQ segment queue via an RDMA transaction through hardware, increment the RDMA counter, and wake up the server.

[0080] Upon wake-up, the server can check the semaphore's value, and if data has entered the queue, the server can process all data in the queue (via RDMA) and reset the RDMA counter to "0." If the semaphore is still in the locked state (as reflected by the semaphore's value), it indicates that the RDMA counter has reacted to some other transaction in the memory region, and the server can return to the sleep state.

[0081] Thus, nodes in a distributed system, such as FAMfs, can use a cluster-wide RBQ for all intra-node communication. Such a cluster-wide RBQ can be implemented as a partitioned segment that is allocated across and by multiple nodes in the distributed system (e.g., server nodes, dispatcher nodes, or I / O nodes in a FAMfs).

[0082] Example method for facilitating shared atomic data

[0083] Figure 5A A flowchart 500 illustrating a method of facilitating LFA-based shared memory access in a distributed system in accordance with an aspect of the present application is presented. During operation, this system allocates a plurality of memory portions including shared remote memory content in a distributed system including a plurality of nodes (operation 502). This system registers the allocated portions with an operating system for access via remote direct memory access (operation 504). This system accesses the allocated portions by a first node to obtain a local copy of the shared remote memory content (operation 506). This system performs an atomic operation on one or more bits of the shared remote memory content via a library facility atomic (LFA) application programming interface (API) call by performing one or more of the following operations (operation 508). This system updates the one or more bits of the shared remote memory content based on a new value and an offset (operation 510). This system fetches a current value of the one or more bits prior to the update from the shared remote memory content based on the offset (operation 512). This system performs an action on the shared remote memory content based on a comparison of the current value with an expected value in the local copy (operation 514).

[0084] Figure 5BA flow diagram 520 illustrating a method that facilitates LFA-based shared memory access in a distributed system in accordance with an aspect of the present application is presented. During operation, this system allocates a plurality of memory portions including shared remote memory content in a distributed system including a plurality of nodes (operation 522). The system registers the allocated portions with an operating system for access via remote direct memory access (operation 524). The system accesses the allocated portions by a first node to obtain a local copy of the shared remote memory content (operation 526). The system performs an atomic operation on the local copy by updating shared objects in the local copy corresponding to one or more bits of the shared remote memory content based on a new value and an offset (operation 528). The system fetches current values of the one or more bits from the shared remote memory content based on the offset (operation 530). The system compares the fetched current values to expected values in the local copy (operation 532).

[0085] If the current values do not match the expected values (decision 534), then the system receives an error message and resolves the error (e.g., to resolve a conflict) (operation 536). If the current values match the expected values (decision 534), then the system performs an atomic operation on the one or more bits of the shared remote memory via a library-structured atomic (LFA) application programming interface (API) call (operation 538). The operation continues at marker A of Figure 6A or marker B of Figure 6B .

[0086] Example method for accessing files and using shared ring buffer queue

[0087] Figure 6AA flowchart 600 illustrating a method of facilitating access to a file in a file system using LFA-based shared memory access in a distributed system in accordance with an aspect of the present application is presented. Remote shared memory content includes a content table of files and one or more counter arrays, an allocated memory portion includes a partition of the content table and the one or more counter arrays, and an entry corresponding to a file in the content table includes: a reference count indicating a number of processes that currently have the corresponding file open; and an offset to an element in one of the counter arrays (operation 602). The system receives a request to open a file having an associated file identifier in a distributed system (operation 604). The shared remote memory content can be accessed through a file system (e.g., FAMfs) associated with the distributed system. The system determines a partition of the content table that includes the file identifier (operation 606). As described above, a client can first perform a lookup in its local copy of the TOC to search for the file identifier without taking a spinlock. If the FileID is not present in the local TOC (not shown), the system obtains a spinlock for the partition of the content table (operation 608). The system searches the partition of the content table for an entry corresponding to the file identifier, where the entry includes: a first reference count; and a first offset to an element in a first counter array, where the element includes a count of actions associated with the file identifier (operation 610). In response to finding the entry, the system increments the first reference count (operation 612). The system releases the spinlock for the partition of the content table (operation 614) and accesses the first counter array at the first offset to increment the count of actions (operation 616). Operation returns.

[0088] Figure 6B A flowchart 620 illustrating a method of facilitating transmitting a message via a shared ring buffer queue using LFA-based shared memory access in a distributed system in accordance with an aspect of the present application is presented. Remote shared memory content includes a global ring buffer queue, an allocated memory portion includes a segment of the global ring buffer queue, and the respective segment includes: a queue for storing data including the message; an input semaphore and an output semaphore for synchronizing access to the queue; and an input pointer and an output pointer for tracking a location related to an insert and remove operation performed on the queue (operation 622). The system accesses, by a first client node, a respective segment of the global ring buffer queue including the remote shared memory content (operation 624). The system acquires the input semaphore by decrementing the input semaphore (operation 626). In response to successfully acquiring the input semaphore, the system increments the input pointer (operation 628). The system sends data via RDMA to a memory location pointed to by the input pointer (operation 630). The system releases the output semaphore by incrementing the output semaphore (operation 632), and operation returns. Figure 6Ccontinues at marker C.

[0089] Figure 6C A flow diagram 640 illustrating a method that facilitates communicating messages using LFA-based shared memory access via a shared ring buffer queue in a distributed system in accordance with an aspect of the present application is presented. The system accesses a respective segment by a first server node to which the respective segment is assigned (operation 642). The system detects that a value of an output semaphore is greater than zero (operation 644). The system acquires the output semaphore by decrementing the output semaphore (operation 646). The system fetches data stored at a memory location pointed to by an output pointer via RDMA (operation 648). The system increments the output pointer (operation 650) and releases the input semaphore by incrementing the input semaphore (operation 652). Operation returns.

[0090] Example computer system and apparatus

[0091] Figure 7 An exemplary computer system 700 that facilitates dynamic allocation of space in a distributed file system in accordance with an aspect of the present application is shown. The computer system 700 includes a processor 702, a volatile memory 706, and a storage device 708. In some aspects, the computer system 700 can include a controller 704 (indicated by dashed lines). The volatile memory 706 can include, for example, a random access memory (RAM), which acts as managed memory and can be used to store one or more memory pools. The storage device 708 can include persistent storage that can be managed or accessed via the processor 702 (or the controller 704). Further, the computer system 700 can be coupled to peripheral input / output (I / O) user devices 710, such as a display device 711, a keyboard 712, and a pointing device 714. The storage device 708 can store an operating system 716, a content processing system 718, and data 736.

[0092] The content processing system 718 can include instructions that, when executed by the computer system 700, can cause the computer system 700 or the processor 702 to perform the methods and / or processes described in the present disclosure. In particular, the content processing system 718 can include instructions for receiving and sending data packets associated with LFA API calls (communication module 720).

[0093] The content processing system 718 can further include instructions to allocate a plurality of memory portions comprising shared remote memory content in a distributed system comprising a plurality of nodes (memory portion allocation module 722). The content processing system 718 can include instructions to register the allocated portions with an operating system for access via remote direct memory access (RDMA) (memory portion allocation module 722). The content processing system 718 can include instructions to access the allocated portions by a first node to obtain a local copy of the shared remote memory content (portion attach module 724). The content processing system 718 can also include instructions to perform an atomic operation on one or more bits of the shared remote memory content via a library-structured atomic (LFA) application programming interface (API) call (LFA API call module 726).

[0094] The content processing system 718 can additionally include instructions to update one or more bits of the shared remote memory content based on a new value and an offset (LFA API call module 726 and bitmap management module 730). The content processing system 718 can include instructions to fetch a current value of the one or more bits prior to the update from the shared remote memory content based on the offset (LFA API call module 726 and bitmap management module 730). The content processing system 718 can include instructions to perform an action on the shared remote memory content based on a comparison of the fetched current value to an expected value in the local copy (LFA API call module 726 and bitmap management module 730).

[0095] The content processing system 718 can further include instructions to access a table of contents and one or more counter arrays as described above with respect to Figure 6A The content processing system 718 can include instructions to access a segment in a global ring buffer queue as described above with respect to Figure 6B and Figure 6C The content processing system 718 can include instructions to access a segment in a global ring buffer queue as described above with respect to

[0096] Data 736 can include any data needed as input or generated as output by the methods and / or processes described in this disclosure. In particular, data 736 can store at least: a virtual copy of a global data structure; a local copy of a global data structure; an indicator of a segment, partition, blob, or allocated memory portion; an indicator or identifier of a node, server node, or client node; an LFA API call; an atomic operation; an arithmetic operation; a logical operation; a value; a current value, expected value, or new value; one or more bits; a bitmap data structure; an entry; a content table; a file identifier; a reference count; an offset; a spinlock; a global ring buffer queue; a queue; a message queue; data; atomic data; encoded or decoded data; tiered data; a state; a word; a shared object; message data; a value of an input pointer, output pointer, input semaphore, or output semaphore; an indicator of a file system or other distributed system; an indicator of a FAM module or other NVMe device; a segment bitmap; and a segment map.

[0097] Figure 8 An exemplary apparatus 800 that facilitates dynamic allocation of space in a distributed file system is shown in accordance with an aspect of the present application. Apparatus 800 can include a plurality of units or devices that can communicate with one another via wired, wireless, quantum optical, or electrical communication channels. Apparatus 800 can be implemented using one or more integrated circuits, and can include fewer or more units or devices than those shown in FIG. 8. Moreover, apparatus 800 can be integrated in a computer system, or implemented as a separate device or devices capable of communicating with other computer systems and / or devices. Figure 8

[0098] Apparatus 800 can also include a non-volatile storage system or memory management unit. Apparatus 800 can include modules or units 802-816 configured to perform functions or operations similar to those of modules 720-734 of computer system 700, including: a communication unit 802; a memory portion allocation unit 804; a portion attachment unit 806; an LFA API call unit 808; a local copy management unit 810; a bitmap management unit 812; a TOC management unit 814; and a queue management unit 816. Figure 7

[0099] ​​In general, the disclosed aspects provide a system that facilitates LFA-based lock-free cluster-wide shared memory access APIs in a distributed system. In an aspect, the system allocates a plurality of memory portions comprising shared remote memory content in a distributed system comprising a plurality of nodes. The system registers the allocated portions with an operating system for access via remote direct memory access (RDMA). The system accesses the allocated portions by a first node to obtain a local copy of the shared remote memory content. The system performs an atomic operation on one or more bits of the shared remote memory content via a library fabric atomic (LFA) application programming interface (API) call, comprising one or more of: updating the one or more bits of the shared remote memory content based on a new value and an offset; fetching a current value of the one or more bits prior to the update from the shared remote memory content based on the offset; and performing an action on the shared remote memory content based on a comparison of the fetched current value to an expected value in the local copy.

[0100] In a variation of this aspect, the system performs the action on the shared remote memory content by receiving an error message in response to determining that the current value does not match the expected value. In response to determining that the current value matches the expected value, the system performs the atomic operation on the one or more bits of the shared remote memory content, wherein the offset comprises a global offset in the shared remote memory content associated with the one or more bits.

[0101] In a further variation of this aspect, the system performs the atomic operation on the local copy by updating a shared object in the local copy corresponding to the one or more bits of the shared remote memory content based on the new value and the offset.

[0102] In a further variation, the atomic operation comprises one or more of: performing an arithmetic operation on the one or more bits, comprising one or more of an addition operation and a subtraction operation; and performing a logical bitwise operation on the one or more bits, comprising one or more of an AND operation, an OR operation, and an exclusive OR (XOR) operation.

[0103] In a further variation, the plurality of nodes comprises a server node and a client node, the server node allocates the plurality of memory portions, and the first node is the client node or the server node. The client node performs the atomic operation by appending to a shared object in the first allocated portion corresponding to the one or more bits based on the new value and the offset.

[0104] In another variant of this aspect, the shared remote memory content is accessed through a file system associated with the distributed system, and the file system is a structured attached memory file system (FAMfs). The server nodes are I / O nodes in the FAMfs. Allocating the plurality of memory portions is performed by an allocator module associated with each I / O node in the FAMfs, and the remotely shared memory content includes a bitmap data structure that tracks the status of physical segments in the memory of the FAMfs, such as used or free.

[0105] In a further variant, the remotely shared memory content includes a file content table and one or more counter arrays, and the allocated memory portions include a partition of the content table and the one or more counter arrays. An entry in the content table corresponding to a file includes: a reference count indicating a number of processes that currently have the corresponding file open; and an offset of an element in one of the counter arrays.

[0106] In a further variant of this aspect, the system receives a request to open a file having an associated file identifier in the distributed system. The system determines a partition of a content table that includes the file identifier. The system obtains a spinlock of the partition of the content table. The system searches the partition of the content table for an entry corresponding to the file identifier. The entry includes: a first reference count; and a first offset to an element in a first counter array, where the element includes a count of actions associated with the file identifier. In response to finding the entry, the system increments the first reference count. The system releases the spinlock of the partition of the content table and accesses the first counter array at the first offset to increment the count of actions.

[0107] In another variant of this aspect, the remotely shared memory content includes a global ring buffer queue, and the allocated memory portions include segments of the global ring buffer queue. A respective segment includes: a queue for storing data including messages; an input semaphore and an output semaphore for synchronizing access to the queue; and an input pointer and an output pointer for tracking positions related to insert and remove operations performed on the queue.

[0108] In a further variant, the system accesses the respective segment by the first client node by performing the following operations. The system acquires the input semaphore by decrementing the input semaphore. In response to successfully acquiring the input semaphore, the system increments the input pointer. The system sends data to a memory location pointed to by the input pointer via RDMA, and the system releases the output semaphore by incrementing the output semaphore.

[0109] In a further variation, the system accesses the respective segment by the first server node to which the respective segment is assigned by performing the following operations. The system detects that the value of the output semaphore is greater than zero. The system acquires the output semaphore by decrementing the output semaphore and takes the data stored at the memory location pointed to by the output pointer via RDMA. The system increments the output pointer and releases the input semaphore by incrementing the input semaphore.

[0110] The data structures and code described in this detailed description are typically stored in a computer readable storage medium, which can be any device or medium that can store code and / or data for use by a computer system. Computer readable storage media includes, but is not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs), DVDs (digital versatile discs or digital video discs), or other media capable of storing computer readable media now known or later developed.

[0111] The methods and processes described in the detailed description section can be embodied as code and / or data, which can be stored in a computer readable storage medium as described above. When a computer system reads and executes the code and / or data stored on the computer readable storage medium, the computer system performs the methods and processes embodied as data structures and code and stored within the computer readable storage medium.

[0112] Moreover, the methods and processes described above can be included in hardware devices or apparatuses. For example, hardware devices or apparatuses can include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), dedicated or shared processors for a particular use, and other programmable logic devices now known or later developed. When the hardware devices or apparatuses are activated, they perform the methods and processes included within them.

[0113] The foregoing description of various aspects has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the various aspects described herein to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. Additionally, the above disclosure is not intended to limit the various aspects described herein. The scope of the various aspects described herein is defined by the appended claims.

Claims

1. A computer-implemented method, comprising: In a distributed system comprising multiple nodes, multiple memory portions are allocated, including shared remote memory content. Register the allocated portion with the operating system for access via Remote Direct Memory Access (RDMA); Access the allocated portion through the first node to obtain a local copy of the shared remote memory content; as well as Atomic operations are performed on one or more bits of the shared remote memory content via a Library Structure Atom (LFA) application programming interface (API) call, the atomic operations including: Update one or more bits of the shared remote memory content based on the new value and offset; Based on the offset, obtain the current value of the one or more bits prior to the update from the shared remote memory content; and Based on the comparison between the obtained current value and the expected value in the local copy, an action is performed on the shared remote memory content; The plurality of nodes includes server nodes and client nodes. The server node allocates the multiple memory portions. The first node is either a client node or a server node. The client node performs the atomic operation by attaching it to a shared object corresponding to the one or more bits in the first allocation portion based on the new value and the offset; The shared remote storage content is accessed through a file system associated with the distributed system. The file system mentioned above is the FAMfs structured secondary storage file system. The server node is the I / O node in the FAMfs. The allocation of the multiple memory portions is performed by an allocator module associated with each I / O node in the FAMfs, and The shared remote memory content includes a bitmap data structure that tracks the status of physical segments in the FAMfs' memory: used or idle.

2. The method as described in claim 1, wherein, Performing the action on the shared remote memory content includes: In response to determining that the current value does not match the expected value, an error message is received; and The offset includes a global offset associated with the one or more bits in the shared remote memory content.

3. The method of claim 1, further comprising: The atomic operation is performed on the local copy by updating the shared object corresponding to one or more bits of the shared remote memory content in the local copy based on the new value and the offset.

4. The method as described in claim 1, in, The shared remote storage content includes a file's content table and one or more counter arrays. The allocated memory portion includes partitions of the content table and the one or more counter arrays, and The entries corresponding to the files in the content table include: A reference count indicating the number of processes currently having the corresponding file open; and The offset from an element in one of the counter arrays.

5. The method of claim 4, further comprising: Receive a request to open a file with an associated file identifier in the distributed system; Determine the partition of the content table that includes the file identifier; Obtain the spinlock of the partition of the content table; Search the partitions of the content table to obtain entries corresponding to the file identifier, wherein the entries include: First reference count; and For a first offset of an element in a first counter array, wherein the element includes a count of actions associated with the file identifier; In response to finding the entry, increment the first reference count; Release the spinlock of the partition of the content table; and Access the first counter array at the first offset to increment the count of the action.

6. The method as described in claim 1, in, The shared remote memory contents include a global circular buffer queue. The allocated memory portion includes a segment of the global circular buffer queue, and The corresponding segments include: A queue is used to store data, including messages. Input and output semaphores are used to synchronize access to the queue; and Input and output pointers are used to track the positions associated with insertion and removal operations performed on the queue.

7. The method of claim 6, further comprising the first client node accessing the corresponding segment in the following manner: The input signal quantity is obtained by decrementing the input signal quantity; In response to successfully acquiring the input semaphore, increment the input pointer; Data is sent to the memory location pointed to by the input pointer via RDMA; as well as The output signal quantity is released by incrementing the output signal quantity.

8. The method of claim 6, further comprising accessing the corresponding segment via a first server node that has allocated the corresponding segment in the following manner: The value of the output semaphore was detected to be greater than zero; The output signal quantity is obtained by decreasing the output signal quantity; Data stored at the memory location pointed to by the output pointer is obtained via RDMA; Increment the output pointer; as well as The input signal is released by incrementing the input signal quantity.

9. A computer system, comprising: processor; as well as A memory coupled to the processor and storing instructions that, when executed by the processor, cause the processor to perform a method, the method comprising: In a distributed system comprising multiple nodes, multiple memory portions are allocated, including shared remote memory content. Register the allocated portion with the operating system for access via Remote Direct Memory Access (RDMA); Accessing the allocated portion via the first node to obtain a local copy of the shared remote memory content; and Atomic operations are performed on one or more bits of the shared remote memory content via a Library Structure Atom (LFA) application programming interface (API) call, the atomic operations including: Update one or more bits of the shared remote memory content based on the new value and offset; Based on the offset, obtain the current value of the one or more bits prior to the update from the shared remote memory content; and Based on a comparison of the current value obtained with the expected value in the local copy, an action is performed on the shared remote memory content; The plurality of nodes includes server nodes and client nodes. The server node allocates the multiple memory portions. The first node is either a client node or a server node. The client node performs the atomic operation by attaching it to a shared object corresponding to the one or more bits in the first allocation portion based on the new value and the offset. The shared remote storage content is accessed through a file system associated with the distributed system. The file system mentioned above is the FAMfs structured secondary storage file system. The server node is the I / O node in the FAMfs. The allocation of the multiple memory portions is performed by an allocator module associated with each I / O node in the FAMfs, and The shared remote memory content includes a bitmap data structure that tracks the status of physical segments in the FAMfs' memory: used or idle.

10. The computer system of claim 9, wherein, Performing the action on the shared remote memory content includes: In response to determining that the current value does not match the expected value, an error message is received; and In response to determining that the current value matches the expected value, the atomic operation is performed on the one or more bits of the shared remote memory content. The offset includes a global offset associated with the one or more bits in the shared remote memory content.

11. The computer system of claim 9, wherein, The method further includes: The atomic operation is performed on the local copy by updating the shared object corresponding to one or more bits of the shared remote memory content in the local copy based on the new value and the offset.

12. The computer system as described in claim 9, in, The shared remote storage content includes a file's content table and one or more counter arrays. The allocated memory portion includes partitions of the content table and the one or more counter arrays, and The entries corresponding to the files in the content table include: A reference count indicating the number of processes currently having the corresponding file open; and The offset of an element in one of the counter arrays.

13. The computer system of claim 12, wherein, The method further includes: Receive a request to open a file with an associated file identifier in the distributed system; Determine the partition of the content table that includes the file identifier; Obtain the spinlock of the partition of the content table; Search the partitions of the content table to obtain entries corresponding to the file identifier, wherein the entries include: First reference count; and For a first offset of an element in a first counter array, wherein the element includes a count of actions associated with the file identifier; In response to finding the entry, increment the first reference count; Release the spinlock of the partition of the content table; and Access the first counter array at the first offset to increment the count of the action.

14. The computer system as described in claim 9, in, The shared remote memory contents include a global circular buffer queue. The allocated memory portion includes a segment of the global circular buffer queue, and The corresponding segment includes A queue is used to store data, including messages. Input and output semaphores are used to synchronize access to the queue; and Input and output pointers are used to track the positions associated with insertion and removal operations performed on the queue.

15. The computer system of claim 14, wherein, The method also includes the first client node accessing the corresponding segment in the following manner: The input signal quantity is obtained by decrementing the input signal quantity; In response to successfully acquiring the input semaphore, increment the input pointer; Data is sent to the memory location pointed to by the input pointer via RDMA; as well as The output signal quantity is released by incrementing the output signal quantity.

16. A non-transitory computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform a method, the method comprising: In a distributed system comprising multiple nodes, multiple memory portions are allocated, including shared remote memory content. Register the allocated portion with the operating system for access via Remote Direct Memory Access (RDMA); Access the allocated portion through the first node to obtain a local copy of the shared remote memory content; as well as Atomic operations are performed on one or more bits of the shared remote memory content via a Library Structure Atom (LFA) application programming interface (API) call, the atomic operations including: Update one or more bits of the shared remote memory content based on the new value and offset; Based on the offset, obtain the current value of the one or more bits prior to the update from the shared remote memory content; and Based on a comparison of the current value obtained with the expected value on the local copy, an action is performed on the shared remote memory content; The plurality of nodes includes server nodes and client nodes. The server node allocates the multiple memory portions. The first node is either a client node or a server node. The client node performs the atomic operation by attaching it to a shared object corresponding to the one or more bits in the first allocation portion based on the new value and the offset. The shared remote storage content is accessed through a file system associated with the distributed system. The file system mentioned above is the FAMfs structured secondary storage file system. The server node is the I / O node in the FAMfs. The allocation of the multiple memory portions is performed by an allocator module associated with each I / O node in the FAMfs, and The shared remote memory content includes a bitmap data structure that tracks the status of physical segments in the FAMfs' memory: used or idle.

17. The non-transitory computer-readable storage medium as claimed in claim 16, in, The shared remote memory contents include a global circular buffer queue. The allocated memory portion includes a segment of the global circular buffer queue, and The corresponding segments include: A queue is used to store data, including messages. Input and output semaphores are used to synchronize access to the queue; and Input and output pointers are used to track the positions associated with insertion and removal operations performed on the queue.

18. The non-transitory computer-readable storage medium of claim 17, further comprising access to the corresponding segment by a first client node in the following manner: The input signal quantity is obtained by decrementing the input signal quantity; In response to successfully acquiring the input semaphore, increment the input pointer; Data is sent to the memory location pointed to by the input pointer via RDMA; as well as The output signal quantity is released by incrementing the output signal quantity.

19. The non-transitory computer-readable storage medium of claim 17, further comprising accessing the corresponding segment via a first server node that has allocated the corresponding segment in the following manner: The value of the output semaphore was detected to be greater than zero; The output signal quantity is obtained by decreasing the output signal quantity; Data stored at the memory location pointed to by the output pointer is obtained via RDMA; Increment the output pointer; as well as The input signal is released by incrementing the input signal quantity.

Images