METHOD AND SYSTEM FOR LIBFABRIC ATOMICS-BASED UNLOCKABLE CLUSTER-WIDE COMMON STORAGE ACCESS API IN A DISTRIBUTED SYSTEM

Abstract

A computer-implemented method, including: Allocating a multitude of memory sections (130, 140, 150) comprising shared remote memory content in a distributed system comprising a multitude of nodes (110, 120, 138, 148, 158); Registering the allocated sections with an operating system for access via remote storage direct access (RDMA); Accessing the allocated sections by a first node (110) to obtain a local copy (111) of the shared remote memory content; and Performing an atomic operation on one or more bits of the shared remote memory content via libfabric-atomic-, LFA,-Application-Programming-Interface-, API, calls, including: Updating one or more bits of the shared remote memory content based on a new value and an offset; Retrieving a current value of one or more bits from the shared remote memory content based on the pre-update offset; and Performing an action on the shared remote memory content based on a comparison of the retrieved current value with an expected value in the local copy; the multitude of nodes includes server nodes (138, 148, 158) and client nodes (110, 120), where the server nodes allocate the multitude of memory sections, where the first node is either a client node or a server node, where the client nodes perform the atomic operation by connecting to a shared object corresponding to the one or more bits in a process based on the new value and Perform the first assigned section based on the offset, where the shared remote storage content is accessed by a file system connected to the distributed system, where the file system is a Fabric Attached Memory Filesystem, FAMfs, where the server nodes are I / O nodes in the FAMfs, where the allocation of the multitude of memory sections is performed by allocation modules connected to each I / O node in the FAMfs, and wherein the shared remote memory content comprises a bitmap data structure that tracks a status, used or free, of physical extents in the FAMf's memory.

Description

BACKGROUND area

[0001] This disclosure relates generally to the field of administration. More specifically, this disclosure relates to a method and a system for libfabric-atomics-based lockless cluster-wide shared memory access API in a distributed system.

[0002] US 9 875 182 B1 concerns systems and procedures for writing data.

[0003] US 2017 / 0 075 856 A1 concerns distributed, lockless RDMA-based memory allocation and release.

[0004] US 2003 / 0217058A1 applies generally to data storage systems and specifically to file systems that support the sharing of storage devices by multiple clients.

[0005] Grun, P. [et al.]: A Brief Introduction to the OpenFabrics Interfaces: A New Network API for Maximizing High Performance Application Efficiency. In: 2015 IEEE 23rd Annual Symposium on High Performance Interconnects, Santa Clara, CA, USA, 26-28 August 2015, pp. 34-39. DOI: 10.1109 / HOTI / 2015.19 concerns a new network API for maximizing the efficiency of high-performance applications.

[0006] The present invention is defined by independent claims 1, 9 and 16. Embodiments are the subject of the respective dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS Fig. shows a diagram of an exemplary environment for enabling libfabric-atomics (LFA) based shared memory access according to one aspect of the present application. Fig. shows a diagram of an exemplary environment for enabling LFA-based shared memory access, including a shared bitmap, according to one aspect of the present application. Fig. shows a diagram of a table of contents, a counter array and exemplary communications associated with accessing files in a file system, in accordance with an aspect of the present application. Fig. shows an exemplary environment for enabling the use of a shared ring buffer queue for message transmission using LFA-based shared memory access in a distributed system according to one aspect of the present application. Fig. shows a flowchart illustrating a procedure that enables LFA-based shared memory access in a distributed system in accordance with one aspect of the present application. Fig. shows a flowchart illustrating a procedure that enables LFA-based shared memory access in a distributed system in accordance with one aspect of the present application. Fig. shows a flowchart illustrating a procedure that enables access to files in a file system using LFA-based shared memory access in a distributed system in accordance with one aspect of the present application. Fig. shows a flowchart illustrating a procedure that enables the transmission of messages over a shared ring buffer queue using LFA-based shared memory access in a distributed system according to one aspect of the present application. Fig. shows a flowchart illustrating a procedure that enables the transmission of messages over a shared ring buffer queue using LFA-based shared memory access in a distributed system according to one aspect of the present application. Fig. shows an exemplary computer system that enables LFA-based shared memory access according to one aspect of the present application. Fig. shows an exemplary device that enables LFA-based shared memory access according to one aspect of the present application.

[0007] In the illustrations, identical numbers refer to the same elements of the illustration. DETAILED DESCRIPTION

[0008] The following description is intended to enable the person skilled in the art to produce and use the aspects and examples and is given in connection with a specific application and its requirements. Various modifications of the disclosed aspects are readily apparent to the person skilled in the art, and the general principles defined herein can be applied to other aspects and applications without departing from the spirit and scope of the present disclosure. Therefore, the aspects described here are not limited to those shown but have the broadest possible scope consistent with the principles and features disclosed herein.

[0009] A crucial task in a distributed system or cluster with multiple components on multiple devices (e.g., high-performance computing (HPC) applications in large-scale clustered computing environments) is the synchronization of access to cluster-wide shared variables and data structures. When implementing a common application programming interface (API) for data access, a balance must be struck between ensuring coherence across all nodes and minimizing additional latency when accessing a shared entity for a read, write, or update operation.

[0010] Some current cluster synchronization methods incorporate a distributed lock manager (DLM) and use a message-based system, for example, between a client and a server. However, as the number of nodes increases (e.g., in a high-performance computing (HPC) environment), the increased latency (from sending, receiving, processing, and responding to messages, including acquiring and releasing locks) can become increasingly unacceptable. Scalability can also be a concern. Increased load on a listening server, combined with an increase in system size, can lead to a bottleneck. Other methods may use a shared message queue, but the performance of such systems can be limited due to implementation complexity and the unavoidable bidirectional traffic.

[0011] The aspects described here offer a system that addresses these challenges by providing cluster-wide lock-free atomic operations using libfabric atomics (LFA) operations on shared memory. In a distributed system or cluster with many nodes, each node can access shared data without acquiring locks by simply using its local copy of a shared object to predict the outcome of a desired update and ensuring the consistency of that update with a compare-and-swap (CAS) libfabric operation. The system can algorithmically distribute the target address of the update across the remote memory of multiple nodes, thereby controlling the collision rate. This reduces both the latency of operations and the utilization of shared resources.The described system can be used to implement various types of shared objects, such as counters, bitmaps, structured data, and cluster-wide synchronization primitives like spinlocks, semaphores, and ring buffer queues. Shared atomic data

[0012] In a distributed system, such as a cluster file system like the Fabric Attached Memory Filesystem (FAMfs), a key challenge is keeping certain important data structures in memory and sharing them across all nodes actively using the file system. One solution is to store all file system metadata in a shared coherent database. However, as the number of nodes increases (as in a high-performance computing (HPC) environment), the performance of such a shared coherent database can decrease exponentially. In the approaches described here, the metadata is still managed in a database and loaded into memory at startup, where it is subsequently manipulated via libfabric-atomics (LFA) API calls. The data structures accessed by these LFA API calls may only remain relevant while the FAMfs is active.The data structures can be rebuilt from the database on every restart. Furthermore, the system can perform all relevant database updates in the background using the FAMfs software, ensuring that these updates do not affect access to the shared metadata within the data structures via LFA API calls. Since the database is never in the path of the LFA operations, there is no additional latency.

[0013] When using LFA calls, the units can be logically termed similarly to a client-server model, although there are no servers in the traditional sense. In the aspects described here, a "server" or "server node" can simply declare or allocate a specific area or portion ("blob") of its own memory as a shared object (which can have any structure). The server node can register the allocated portions with the operating system, informing the operating system that the allocated area in memory needs to be locked. The server node can also generate access keys for the "clients" or "client nodes." This concludes the server's responsibility. The remaining operations can be performed by the "clients," as described below. Fig. described. It should be noted that these clients and servers can be the same node / process or different nodes / processes.

[0014] The system can operate with the shared objects of the allocated blobs as a single contiguous space consisting of all allocated blobs. This single contiguous space can be referred to as an "LFA space" or "shared remote memory content." The described system can comprise multiple LFA spaces, all of which are independent of each other; that is, LFA blobs allocated by different groups of server nodes in a distributed system. The size of each blob allocated by a server can be different and therefore may not be the same as the size of the other blobs in a given LFA space.

[0015] To access shared data in a specific LFA scope, a node must connect to the LFA scope via an LFA API call, establishing the necessary libfabric connections to all servers within that scope. LFA API calls can operate using global offsets within the given LFA scope. Using a specific offset, the LFA API can calculate the desired location in remote memory—that is, the libfabric address within the server that has mapped a portion of its memory as global space, as well as a local offset within the given LFA scope. The system can resolve the server address based on a fast binary search in an ordered array of registered LFA blobs.

[0016] The consistency of libfabric-atomics operations can only be ensured through the use of libfabric calls. Consequently, all updates to content within the LFA scope must be performed via libfabric calls. For example, a client can only access the LFA scope via an LFA API call, even if a target object or memory location resides in or is located within the same client's memory.

[0017] A target object is a shared object that can be a 32-bit or 64-bit word with a specific global offset within an LFA range. The system can support various operations for both 32-bit and 64-bit words. Bulk get / put operations allow loading and writing remote memory to and from the local buffer. These operations do not guarantee consistency and should only be used when concurrency is not a concern, such as during startup when the system populates the LFA ranges from the database.

[0018] The system can support simple arithmetic operations (e.g., adding and subtracting a value from a remote location) as well as logical bitwise operations (e.g., logical AND, OR, and XOR operations). Furthermore, the system can support compare-and-swap operations, where it can compare a retrieved remote value with a local value and replace the remote value with a new one if it finds that the retrieved remote value matches the local value. The arithmetic and bitwise operations can support the "do-and-fetch" variant. Additionally, atomic operations or instructions can be implemented in hardware.

[0019] Since each client maintains its own complete local copy of the shared remote memory content, each client can browse its local copy in the normal programmatic way without having to lock anything. When updating the shared remote memory content, the client can make an LFA API call to ensure that the update is valid and coherent (e.g., through a do-and-fetch operation). If a collision occurs (i.e., if more than one node has attempted to update the same word in remote memory simultaneously), it is the client's responsibility to resolve that collision. In practice, in a highly parallel distributed environment, the probability of a collision may be low, highlighting the concept that in some cases it may be simpler or more efficient to ask for forgiveness than for permission.

[0020] Fig. Figure 100 shows a diagram of an exemplary environment 100 for enabling shared atomic data, in accordance with an aspect of the present application. The environment 100 may include: several “server” nodes, e.g., a Server-Node_0 138, a Server-Node_1 148, and a Server-Node_N 158; and several client nodes, e.g., a Client-Node_0 110 and a Client-Node_M 120 (“Client Node 110 and 120”).

[0021] Each server node can allocate a "blob" of its local memory and register the allocated blob with the operating system so that it can be accessed via Remote Memory Access (RDMA). For example, server node_0 138 can allocate an LFA blob_0 130, server node_1 148 an LFA blob_1 140, and server node_N 158 an LFA blob_N 150. Each blob can contain a variety of bits, sets of one or more bits, or words. For example, LFA blob_0 130 can contain at least: one word W[0] 132; one word W[1] 134 and one word W[2] 136. Similarly, LFA-Blob_1 140 can contain at least one word W[i] 142 and LFA-Blob_N 150 can contain at least one word W[n] 152. The system can thus create an "LFA area" or "shared remote memory content" that contains these allocated LFA blobs (130, 140 and 150).

[0022] Each client can then connect to the LFA space to obtain its own local copy of the shared remote memory content. For example, each of client nodes 110 and 120 can connect to the LFA space, which consists of LFA blobs 130, 140, and 150, to obtain its own local copy of the shared remote memory content (e.g., a local copy 111 for client node_1 110 and a local copy 121 for client node_M 120). Client nodes 110 and 120 can then load the shared remote memory content in the LFA space into their respective local buffers using an `lfa_get()` call and begin working on the obtained local copy in their buffers.

[0023] Local copy 111 can contain at least the following: one word W[0] 112; one word W[1] 114; one word W[i] 116; and one word W[n] 118. Local copy 121 can contain at least the following: one word W[0] 122; one word W[1] 124; one word W[i] 126; and one word W[n] 128. The words in each of the local copies can correspond to the words in each of the LFA blobs 130, 140, and 150. For example, the word W[0] 122 in local copy 121 can correspond to the word W[0] 132 of LFA blob 130 (and can be called / updated in the LFA space via communication 160). The word W[i] 126 in local copy 121 can correspond to the word W[i] 142 of LFA blob 140 (and can be called / updated via communication 162); and the word W[n] 128 in local copy 121 can correspond to the word W[n] 152 of LFA blob 150 (and can be called / updated via communication 164).

[0024] When a client node wants to update a specific word within the LFA scope, the system can issue an LFA API call (e.g., a libfabric operation) based on a new value for the specific word and a global offset for that word within the LFA scope, for example, by using the new value and the global offset as arguments for the LFA API call. The system can determine a target server associated with the LFA blob where the specific word is stored, using a local offset within the target server's remote memory, and then perform the atomic libfabric operation on the specific word (e.g., a "desired action" by replacing an existing value with a new one).

[0025] If the LFA API call is a "do-and-fetch" operation, the LFA API call can alternatively retrieve the current value of the word in question from the remote memory of the target server before performing the desired action on that word. To verify that no collision occurs, the system can compare the retrieved current value in remote memory with the expected value in the local copy (e.g., a "local current value"). If the queried current value does not match the expected value, the client can take appropriate action, such as resolving the collision. Shared Bitmap

[0026] Bitmaps are well-known structures widely used in file systems and other applications that track free / used memory or storage space. Bitmaps can be implemented as a local memory array where a single bit corresponds to a block of data. A bit value of "1" indicates that the block is in use, while a bit value of "0" indicates that the block is free. While bit manipulation can be efficient, as modern CPUs can implement various bit manipulation instructions in hardware, manipulating a single bit in a bitmap shared by multiple nodes in a distributed system (such as an HPC cluster) can be challenging.

[0027] In a Fabric Attached Memory Filesystem (FAMfs), the system can use LFA API calls to implement a shared bitmap that tracks the usage of media extents in an extent bitmap. The FAM address space can be organized into a number of chunks, while a FAM module can be allocated extents. An extent can be a minimal unit of FAM allocation from a single FAM module. FAMfs can organize extents (allocated by different FAM modules) into slabs to enable the implementation of a specific data protection scheme (which can be set at the file level). A slab can be used as a container for stripes (e.g., by serving as a container for stripe allocation). Stripes are collections of data blocks with the same level of protection as the respective container slab. A slab map is a data structure that can describe the geometry and layout of a file system.An "extent bitmap" describes which FAM extents are grouped together to form a specific slab and support a particular protection scheme. It's a data structure that tracks whether a specific extent (on a FAM module or at the non-volatile storage (NVMe) level) is in use or free.

[0028] Both slab maps and extent bitmaps can be common data structures in FAMfs. In FAMfs, the system can implement extent bitmaps as a global bitmap array with LFA-based cluster-wide data structures. These extent bitmaps can support memory allocation across all FAM modules without requiring cluster-wide locks. This global bitmap array can be distributed across all nodes, which then allocate the FAM extents on behalf of FAMfs user processes.

[0029] Certain nodes in FAMfs can make LFA API calls on the extent bitmap data structure. For example, I / O nodes in FAMfs can contain an allocation module that allocates its specific shared memory area (for a slab map or an extent bitmap). These I / O nodes can be referred to as "allocation nodes." Each allocation node can also maintain its own local copy of the full bitmap, as shown below. Fig. As described, each allocation node can be responsible for its own portion of the FAM pool, thus ensuring the locality of the algorithms performing the allocation. However, the product of the allocation—a bitmap of the allocated FAM extents indicating the "used / free" status—can be a global entity shared by all participating nodes (e.g., I / O nodes or allocation nodes in FAMfs). This ensures consistency of allocation information across all participating nodes. In contrast, locking the entire map for each allocation would be prohibitively expensive. Therefore, the current solution provides an efficient way to enable a shared global bitmap array for multiple allocation nodes in a distributed file system (e.g., FAMfs).

[0030] In FAMfs, one extent bitmap is used per FAM module, and extent bitmaps are shared only between the allocation node and any node that needs to access the extent maps for all FAM modules in the FAMfs pool. Consequently, each server creates and allocates an LFA blob for the portion of the global extent map for which it is responsible. Simultaneously, each server joins every LFA blob in the LFA scope, including its own allocated LFA blob. As described here, the system performs all LFA operations via LFA API calls to ensure consistency at the CPU cache level, even when a server wants to access data stored in its own allocated LFA blob. In FAMfs, the system determines the partitioning of the LFA extent map based on the LFA API caller (e.g., the FAMfs Dynamic Space Allocator).

[0031] Fig. Figure 200 shows a diagram of an exemplary environment for enabling shared atomic data, including a shared bitmap, in accordance with an aspect of the present application. The environment 200 can comprise a plurality of nodes: an assignment node_0 (Figure 210); an assignment node_1 (Figure 220); and an assignment node_N (Figure 230). Each assignment node can allocate its own "blob" or "bitmap segment." All allocated blobs or bitmap segments can be virtually or logically combined to form the LFA space. Furthermore, each assignment node can receive its own local copy of all bitmap segments contained in the LFA space.

[0032] For example: Assignment node_0 210 can assign a Bitmap segment_0 212, which can contain a multitude of bits; assignment node_1 220 can assign a Bitmap segment_1 222, which can contain a multitude of bits; and assignment node_N 230 can assign a Bitmap segment_N 232, which can contain a multitude of bits. The bits in these assigned bitmap segments can be represented by the empty square fields. An LFA area 240 can represent a virtual bitmap containing all the bitmap segments (e.g., 212, 222, and 232) assigned by the assignment nodes (e.g., 210, 220, and 230).

[0033] The assignment node_0 210 can obtain a local full copy of bitmap 214 by attaching to the virtual bitmap 240 (e.g., as indicated by arrows 266 and 274), which in turn is attached to each of the bitmap segments 212, 222, and 232 (e.g., as indicated by arrows 252 and 254 for bitmap segment_0 212, arrows 256 and 260 for bitmap segment_1 222, and arrows 262 and 264 for bitmap segment_2 232). Similarly, assignment node_1 220 can obtain a local full bitmap copy 224 by appending to the assigned bitmap segments in a similar manner (e.g., as indicated by arrows 270 and 276). Assignment node_2 230 can also obtain a local full bitmap copy 234 by appending to the assigned bitmap segments (e.g., as indicated by arrows 272 and 278).

[0034] The LFA API calls for the extent bitmap can implement a global find-first-clear-bitand-set procedure that traverses the global bitmap to locate the first unoccupied bit, starting from a specified initial position. This initial position can be determined by an algorithm designed to reduce competition among requesters, i.e., one that attempts to distribute the bits. Since the requesting process always maintains a local copy of the entire bitmap in its local buffer, the system can efficiently search the local buffer first using built-in CPU instructions (e.g., by inverting the value and then performing an ffs() operation in x86). The ideal outcome is that the bit at the initial position is unoccupied. However, if the bit at the initial position is not unoccupied, the ffs() instruction can return the next available position.The API can then perform an atomic logical OR and fetch call on libfabric, which attempts to transfer the new value to the global shared memory.

[0035] If the bit at the corresponding global memory location is unoccupied, the operation completes successfully. If the bit at the corresponding global memory location is not unoccupied, indicating that the bit has been set by another actor or entity, the LFA call can return both an error code and the contents of the global memory that were retrieved immediately before the attempt to set or transfer the bit. This provides a current snapshot of the bitmap's current value, allowing the system to make an intelligent decision based on this information. The system can generally attempt to remain within the local scope of the algorithm, for example, on the same node, which is determined by the initial value calculated by a hash function.This can lead to a reduction in I / O traffic in the fabric and also contribute to a reduction in concurrency. The system can only switch to the LFA area of ​​another node once it determines that all bits in its "local" LFA area are set.

[0036] It is important to note that the "fetch" part of the LFA API call can always be executed. Thus, even if the first attempt to set a bit is successful, the system can return the updated contents or current value of the global memory, including any bits that may have been set by other operations. From the perspective of each allocation node, the extent bitmap can represent a kind of self-healing local copy of the shared global memory. Cluster-wide spinlocks and semaphores

[0037] While LFA API calls can enable lock-free access to shared data (as described above), they are generally limited to operation with separate memory words. In some cases, the system may need to ensure consistent transactional access to multiple fields simultaneously, for example, when a particular data structure needs to be traversed and subsequently extended or updated, maintaining consistency across more than one data word. Because LFA API calls are generally only coherent within the boundaries of a single word, the system must manage access to the entire structure in another way.

[0038] The aspects described here govern access to the entire data structure through the implementation of a global spinlock. A spinlock can be used in the kernels of modern operating systems to enable synchronization between multiple threads. In these aspects, the system can create a global, cluster-wide spinlock based on LFA API calls. When a calling process needs to obtain the spinlock, it can issue an LFA compare-and-swap (CAS) call against one of the predefined LFA domains, which contain locks for various purposes. The CAS call can set a remote location to a specific value, but only if the remote location contains an initial value that matches an expected value (e.g., if the retrieved current remote value matches a local current value). If the queried value matches the expected value, the operation is successful.The calling process can obtain the spinlock and proceed as needed.

[0039] If the requested value does not match the expected value, this means that another process has already acquired the spinlock and is working on the data protected by the spinlock. The calling process then calls the CAS again and continues the loop until either the spinlock is successfully acquired or a timeout period expires. When the calling process (or any other process that acquires the spinlock) has finished updating the data in the critical section, this process can release the spinlock with a simple LFA write call, which can reset the value of the removed spinlock to the unlocked state.

[0040] Similar to the global, cluster-wide spinlock, LFA API calls can implement a cluster-wide counting semaphore, enabling access to countable resources. Instead of the CAS call, the semaphore API can use atomic add-and-fetch and decrement-and-fetch calls. When a new free object is discovered in a pool protected by the semaphore, the system can atomically increment the semaphore's value by one. When a client or other entity needs to acquire an object in the pool protected by the semaphore, the client can attempt to decrement the semaphore by one. If the semaphore's value (after the decrement attempt) is greater than zero, the operation can proceed. If the semaphore's value (after the decrement attempt) is not greater than zero, meaning the semaphore was already zero at the time of the call, the client can retry acquiring the object at a later time.

[0041] Both the global spinlock and global semaphore implementations are based on the LFA space described here, and both synchronization primitives can utilize the LFA infrastructure. For example, the system can store these two synchronization primitives either in a special LFA space separate from other shared data or as part of global structures that contain both the lock and the protected data structure, as described below with respect to Fig. described. Table of contents and counter array

[0042] In large systems, such as HPC applications and distributed systems or clusters, a key task is tracking resource utilization for both statistical performance analysis and actual resource planning and allocation. This can be challenging because HPC applications often access these resources concurrently on a massive scale. The description above refers to implementing atomic counters using the LFA API, but allocating these counters can be difficult in a system where thousands of nodes are simultaneously creating hundreds of files.

[0043] One aspect of the described system provides a simple and reliable mechanism for assigning different usage counters within a cluster. The system provides a Table of Contents (TOC) that lists all existing counters. It also allocates a memory block containing the counters themselves. The system can partition the TOC and the counter block (or blob) in the same way as a regular LFA area to distribute fabric traffic across multiple server nodes. The system can protect each TOC partition with a global spinlock.

[0044] In the FAMfs example, each file has a unique identifier (“FileID”), as described below. A client can perform a TOC search for a specific FileID in a given LFA by first searching its local copy of the TOC without obtaining the global spinlock, in accordance with the general LFA principle. If the given FileID is found in the local copy of the TOC, there is a high probability that the file is already stored at the corresponding global location. The client can then obtain the spinlock on the TOC partition belonging to the specified LFA and verify this. If the file has already been accessed by another process and the global TOC entry is empty or contains a different FileID, the system can perform a full scan of the TOC and assign a new TOC entry.If the specified FileID is not found in the local copy of the TOC, the system can, based on an algorithmic decision, acquire the global spinlock on the TOC partition belonging to the specified LFA. The system can then search the TOC entries for the FileID to find, for example, an empty entry or an entry with the given FileID if it has already been assigned by another client process. Similar to the general principle of LFA design, all counter-related operations are unidirectional; that is, no message exchange takes place between clients and servers. All operations are atomic RDMA transactions from the client to the server node's memory.

[0045] In FAMfs, the file system must track both open files and updates to those open files. FAMfs is a checkpoint and recovery-oriented file system and therefore must also track when files are "laminated," that is, when the erase code (EC) calculation for a file is complete and the file is protected against system, device, or FAM module failures. However, since the EC encoding process is performed on physical data blocks within the FAM modules, not on files, the aspects described here offer an alternative way to track the lamination progress. One solution is to simply allocate a predefined number of counters. However, this can be either wasteful if too many counters are allocated or extremely inefficient if a client process has to wait for a counter to become available.

[0046] In FAMfs, each file has a unique identifier ("FileID") derived from the file's fully qualified name. This FileID can be used to track all counters associated with the file, such as the number of opened files, the number of write operations, etc. When the system makes an `open_file()` call, the client process first looks for the FileID (e.g., "X") in the TOC within a specific LFA range. The TOC LFA range can be shared by all server nodes (as described above in relation to...). Fig. described) and can be protected by a global spinlock (as described below in relation to Fig. (described). Similar to general LFA principles, the system aims to algorithmically distribute the load as much as possible. The system can use the FileID as input for a hash function that can distribute the load evenly across all nodes serving a specific LFA range. The hash function can provide a high probability that multiple open_file() calls from the same client node will be directed to different server nodes, which can result in all I / O nodes in the cluster being loaded evenly. Thus, the hash function can determine, based on the file's unique FileID, that a specific server is responsible for a particular file. The system can translate the filename into its unique FileID, which points to a specific LFA range, similar to the above regarding... Fig. described extent bitmaps.

[0047] Fig. Figure 300 shows a diagram 300 of a table of contents 320, a counter array 340, and exemplary communications associated with accessing files in a file system, in accordance with an aspect of the present application. The diagram 300 may contain a requester 310 (e.g., a client process), a table of contents (TOC) 320, and a counter array 340. The requester 310 may perform a series of steps (311-315) to access a particular file. The table of contents 320 may contain an associated spinlock (SL) 322 and entries that specify at least the following: a file ID 324 for a particular file (e.g., the FileID); a reference counter 326, which may indicate how many clients currently have the particular file open; and an offset 328, which may be an offset in the counter array 340. The counter array 340 can be an array that is assigned to manage the counters for a specific FileID.An entry 330 in TOC 320 can contain a FileID of "0", a reference count of "0", and an offset of "0", which can indicate an empty entry. An entry 332 in TOC 320 can contain a FileID of "X", a reference count of "RC_X", and an offset of "OFF_X". Similarly, an entry 334 in TOC 320 can contain a FileID of "Y", a reference count of "RC_Y", and an offset of "OFF_Y". The reference count and the offset can be integer values.

[0048] If the system wants to open a file with FileID = "X" via a client process, the client process can first compute the TOC to which this FileID belongs, for example, by using the hash function. As described above, the client can first search for the FileID in its local copy of the TOC without activating the global spinlock. If the FileID does not exist in the local TOC, the client process can determine that TOC 320 of a particular LFA (not shown) is the appropriate TOC. The client process can obtain spinlock 322 for TOC 320 (as indicated by a Step_1 311 and an Erlangen-350 communication). The client process can scan TOC 320 for the FileID of "X" (as indicated by a Step_2 312 and a Scan-352 communication). The scan can start at entry 330. If the client process finds the FileID of "X" (i.e.,If a matching entry 332 is found, the system can increment the reference counter value (as indicated by a Step_3 313 and a Reference-Count-Increment-354 communication). If the client process cannot find the FileID of "X" (e.g., if entry 332 does not exist in TOC 320), the system can create an entry for the FileID of "X", increment the reference counter value by one, initialize all other relevant counters, and record a selected offset in counter array 340 in the new TOC entry (e.g., by using the first available element or by some other method of determining an element in counter array 340, where the current value of the counter itself is tracked).

[0049] The client process can then release spinlock 322 (as indicated by a Step_4 314 and a Release-- 356 communication), making TOC 320 available to all other requesting client processes, nodes, or entities. At this point, the client process can now directly access and manipulate the counter in the counter array. For example, the client process can increment the appropriate counter in counter array 340 (as indicated by a Step_5 315 and a Counter-Increment-358 communication). The client process can determine the position of the appropriate counter in counter array 340 by using the offset 328 of the matching entry 332 (as indicated by a dashed arrow 362 from entry 332 to an element 346 in counter array 340 labeled "Counter[j]").

[0050] In this way, once the file is opened, the client process can use one or more counters to determine how many write operations have occurred (a first counter array) and how many write operations have been transferred (a second counter array). The system is not limited to one or two counter arrays but can use as many or as few as needed. The system can wait for a parity thread to calculate the parity for a specific number of blocks corresponding to a file. When the parity calculation for all these blocks is complete, the system can determine that the file is ready for lamination.

[0051] The system can frequently access these counters because processing data blocks can result in accessing and incrementing the corresponding counters. Using LFA API calls and partitioning these TOCs and counter arrays into separate LFA areas can minimize competition for the counters. The latency associated with performing steps 311-314 (in Fig. The connection (TOC lookup) only needs to occur once, when the file is opened. Once the client process has determined the offset, it can directly access the counter without further locking; that is, it can only use LFA API calls in a lockless, cluster-wide manner. This slight delay when opening / closing a file does not significantly impact performance because the system performs many other operations in parallel with the TOC lookup, which can hide or mask this additional latency.

[0052] When the system wants to close the open file via the client process, it calls spinlock 322 again, locates the entry for the requested FileID, and decrements the reference count by one. If the reference count is "0", meaning that no other entities, processes, or nodes in the cluster have the file open, the system can delete the corresponding offset in the entry and the associated element or block in counter array 340. The system can then perform the necessary updates and subsequently release spinlock 322.

[0053] Therefore, a client process does not need to know the actual physical location of a counter for a specific file. Using the steps and operations described above, the client process can simply look up the relevant information in a given TOC and proceed as described above. Global ring buffer queue

[0054] While the aspects described above target atomic operations on specific remote shared memory contents (by eliminating message passing), nodes in an HPC cluster still need to perform message passing (e.g., by using message queues), as this is a fundamental task in any distributed application. Generally, message queue implementations are relatively complex and require a significant amount of resources. The described aspects of the system use a simplified mechanism to provide a shared queue that can quickly and efficiently forward short, fixed-size messages from node to node.

[0055] The system is based on the one-way communication described above for LFA API calls and operations and uses a Ring Buffer Queue (RBQ). The RBQ is a distributed structure divided into "segments" across a multitude of servers, similar to the scope bitmap or the data contained in LFA blobs that comprise an LFA scope. Each RBQ segment can contain: a memory block containing the messages to be transmitted; two semaphores for synchronizing access; and two pointers / counters for tracking the positions of insert and remove operations, as described below. Fig. described.

[0056] Logically, a single shared RBQ can therefore comprise a multitude of partitions or segments, each served by a separate server node, much like each LFA blob is served by a separate server node. The details of this implementation are not visible to clients. A client can simply join a queue by name. The RBQ can include an API that automatically handles the routing of incoming requests.

[0057] Each server in a distributed system (e.g., each I / O or allocation node in FAMfs) can create a segment for a specific queue and also maintain the RBQ scope, which contains or is associated with all the necessary components for the created segment. The system can set the size of the RBQ at startup. The system can also initialize all necessary components during segment or queue creation via the respective server node. The system can create multiple queues, and each node can act as both a server for one RBQ and a client for another. Clients can join existing queues, which may cause the atomic usage counter on the server to be incremented to track overall queue usage and prevent unexpected queue exit behavior.

[0058] Fig. This shows an example environment 400 for enabling the use of a shared ring buffer queue for message delivery, as required by one aspect of this application. The environment 400 can support a variety of clients (e.g., clients C0-C). N ) and comprise a variety of servers, including: a Server_0 410; a Server_1 420; and a Server_K 430. Each server can create an RBQ segment, which can encompass, contain, include, or be a queue. For example: Server_0 410 can create an RBQ Segment_0 412; Server_1 420 can create an RBQ Segment_1 422; and Server_K 430 can create an RBQ Segment_K 432.

[0059] Clients can connect to the RBQ segments to obtain a local copy of the RBQ, similar to how clients can connect to any LFA blob or extent bitmap segment to obtain their respective local copies (as described above regarding the Fig. (described). For example: Client C3 can connect to RBQ-Segment_0 412; Client C7 can connect to RBQ-Segment_1 422; and Client C N can connect to RBQ-Segment_K 432.

[0060] The lower half of Fig. shows a detailed view of the communication between Client C N 450 and Server_K 430. Client C N 450 can contain an RBQ push module 452 that handles access to a specific RBQ segment, and Server_K 430 can contain an RBQ pop module 440 that handles access to the same RBQ segment. RBQ push 452 and RBQ pop 440 can be a module, a process, a component, or an entity implemented in middleware, software, or hardware.

[0061] Server_K 430 can also contain an RBQ segment_K 432, which can include: a memory block 433 (e.g., an array 433 or a queue 433) that can contain data / messages transferred in elements such as W[0] 434, W[1] 436, and W[2] 438; an input semaphore (IS) 462 and an output semaphore (OS) 464 to synchronize access to the queue 433; and an input pointer (IP) 460 and an output pointer (OP) 466 to track positions for inserting and removing messages from the queue 433.

[0062] During initialization and other startup operations, the system can set IS 462 to the maximum size of queue 433. The system can set OS 464 to "0" to indicate that queue 433 contains no data (e.g., messages). The system can also set both IP 460 and OP 466 to "0" as initial values.

[0063] During operation, a client (e.g., Client C) can N Client C (e.g., 450), which wants to place a message in the ring buffer queue (e.g., queue 433), first attempts to obtain IS 462. The value of IS 462 indicates the number of free slots in the queue. If the value of IS 462 is greater than 0, the client C N 450 IS 462 is obtained by decrementing the value of IS 462 by "1" (as indicated by a step 471 (IS--)). If the value of IS 462 is "0" (meaning there are no free or available slots in the RBQ or queue 433), the client must wait until the server has finished processing a request and releases IS 462. Waiting for semaphore availability is described below.

[0064] After successfully obtaining IS 462, the client can increment IP 460 (as indicated by step 472 (IP++)), so that IP 460 now points to W[1] 436 (as indicated by a dashed curved arrow 477) instead of W[2] 438 (as indicated by a bold curved arrow 476). Since the RBQ is a ring buffer, the system can reset IP 460 to "0" when the end of the buffer (i.e., queue 433) is reached. The client C N 450 can send data (e.g., a message) via RDMA to the memory location referenced by IP 460 (as specified in step 473). It should be recalled that each message in RBQ segment_K 432 has the same fixed size. Therefore, the system can determine the location of this memory (according to W[2] 438) by a simple offset-based calculation. Finally, the client C NRelease OS 464 by incrementing the value of OS 464 by "1" (as specified in step 474 (OS++)). This increment informs Server_K 430 that there is data in queue 433 that can be processed.

[0065] Under normal operation, server_K 430 can wait for OS 464, which has the value "0", if queue 433 is empty. If OS 464 is from client C NWhen 450 is incremented (as part of step 474), Server_K 430 can obtain OS 464 by decrementing the value of OS 464 by 1 (as indicated by step 481 (OS--)). Server_K 430 can then use RDMA to retrieve the message stored at the location pointed to by OP 466 (i.e., W[2] 438) (as shown in step 482). After retrieving the message or data from queue 433, Server_K 430 can increment OP 466 by 1 (as shown in step 483 (OP++)), so that OP 466 now points to the next item in queue 433 (as shown by a dashed curved arrow 487) instead of W[2] 438 (as shown by a solid curved arrow 486). This is similar to IP 460 for Client C. NIf 450 is functioning, OP 466 can return to the beginning of the circular RBQ segment when OP 466 reaches the end of queue 433. To avoid tying up valuable resources, Server_K 430 can copy data from global memory to its local buffer (not shown) and then release IS 462 by incrementing its value by 1 (as indicated by a step 484 (IS++)), resulting in a free slot. It's important to note that while the control information for each queue (e.g., IP 460, OP 466, IS 462, and OS 464) can be stored in the LFA areas, the data segments of each queue do not need to be stored in the LFA areas. The data segments can only exist on the "server" node, i.e., for the process that owns a particular queue segment.If the IP / OP state is determined by atomic transactions, as described above, the client can perform a regular RDMA read / write access to / from a corresponding data segment defined by these pointers. Therefore, the client does not need to maintain a local copy of the entire queue, since the atomicity of RDMA is ensured by the queue semaphores in LFA.

[0066] Waiting for a semaphore to become available can be implemented in several ways. One initial implementation for a process seeking a semaphore is to simply forward an atomic CAS call until the value changes. The wait time associated with this spin time is localized. That is, although the system wastes local CPU cycles during the wait, the spin time does not generate traffic in the fabric because the value being checked resides in the memory of the same node. Nevertheless, this method still consumes CPU cycles.

[0067] A second implementation involves using regular messages between the client and the server, for example, to send a notification or a wake-up message to the server after a client has added an item to a previously empty queue. With this implementation, the server can enter a sleep state and wait for a wake-up message once its queue is full. While this solution introduces an initial wake-up latency, it can reduce resource consumption while the system is idle. In a scenario with many messages, the server is not put into a sleep state, no wake-up messages are sent, and the system can operate quickly. However, this implementation still involves an initial wake-up latency.The system can compensate for this initial latency by sending a wake-up message to the server only when a certain or predetermined number of messages are in the queue, instead of sending a wake-up message every time the queue size reaches "1" (or another number smaller than the predetermined number).

[0068] A third implementation is the use of libfabric RDMA counters to enable server wake-up. Some libfabric providers allow the use of "passive" RDMA counters, where a counter can change its value when a specific memory area is accessed (e.g., written or read). Such an implementation can support separate counters for read and write operations. When the server clears its queue, it can put the write counter into a sleep state. A client can then place data or a message into the RBQ segment queue via an RDMA transaction over the hardware, incrementing the RDMA counter and waking the server.

[0069] After waking up, the server can check the semaphore value, and if data has been queued, the server can process all the data in the queue (via RDMA) and reset the RDMA counter to zero. If the semaphore is still locked (reflected by the semaphore value), this means that the RDMA counter has responded to another transaction in the memory space, and the server can return to sleep.

[0070] In a distributed system (such as FAMfs), nodes can use a cluster-wide RBQ for all communication between nodes. This cluster-wide RBQ can be implemented as partitioned segments allocated by and across multiple nodes (e.g., server nodes, allocation nodes, or I / O nodes in FAMfs) in the distributed system. Exemplary procedure for enabling the sharing of atomic data

[0071] Fig. Figure 500 shows a flowchart illustrating a procedure that enables LFA-based shared memory access in a distributed system according to one aspect of the present application. During operation, in a distributed system comprising a multitude of nodes, the system allocates a multitude of memory portions containing shared remote memory contents (Operation 502). The system registers the allocated portions with an operating system to enable direct remote memory access (Operation 504). The system accesses the allocated portions through a first node to obtain a local copy of the shared remote memory contents (Operation 506).The system performs an atomic operation on one or more bits of the shared remote memory content via libfabric-atomic (LFA) application programming interface (API) calls (Operation 508) by executing one or more of the following operations: The system updates the one or more bits of the shared remote memory content based on a new value and an offset (Operation 510). The system retrieves a current value of the one or more bits prior to the update from the shared remote memory content based on the offset (Operation 512). The system performs an action on the shared remote memory content based on a comparison of the current value with an expected value in the local copy (Operation 514).

[0072] Fig. Figure 520 shows a flowchart illustrating a procedure that enables LFA-based shared memory access in a distributed system in accordance with one aspect of the present application. During operation, in a distributed system comprising a multitude of nodes, the system allocates a multitude of memory sections containing shared remote memory contents (Operation 522). The system registers the allocated sections with an operating system to enable access via direct remote memory access (Operation 524). The system accesses the allocated sections through a first node to obtain a local copy of the shared remote memory contents (Operation 526).The system performs an atomic operation on the local copy by updating a shared object in the local copy, based on a new value and an offset, that corresponds to one or more bits of the shared remote memory content (Operation 528). The system retrieves a current value of the one or more bits from the shared remote memory content based on the offset (Operation 530). The system compares the retrieved current value with an expected value in the local copy (Operation 532).

[0073] If the current value does not match the expected value (Decision 534), the system receives an error message and corrects the errors (e.g., to resolve a collision) (Operation 536). If the current value matches the expected value (Decision 534), the system performs the atomic operation on the one or more bits of the shared remote memory via libfabric-atomic (LFA) calls to the application programming interface (API) (Operation 538). The operation is performed either at Label A of Fig. or Label B of Fig. continued. Example procedure for accessing files and using a shared ring buffer queue

[0074] Fig. Figure 600 shows a flowchart illustrating a procedure that enables access to files in a file system using LFA-based shared memory access in a distributed system, in accordance with one aspect of the present application. The remote shared memory content comprises a file index and one or more counter arrays. The allocated memory sections comprise partitions of the file index and the one or more counter arrays. The file index entries corresponding to the files include: a reference counter value indicating the number of processes currently having a corresponding file open; and an offset to an element in one of the counter arrays (Operation 602). The system receives a request to open a file with an associated file identifier in the distributed system (Operation 604). A file system (e.g.,The client (FAMfs), which is associated with the distributed system, can access the shared remote memory content. The system determines a partition of the table of contents (TOC) that contains the file identifier (Operation 606). As described above, the client can first search for the file identifier in its local copy of the TOC without obtaining a spinlock. If the FileID is not present in the local TOC (not shown), the system obtains a spinlock on the partition of the table of contents (Operation 608). The system searches the partition of the table of contents to obtain an entry that corresponds to the file identifier, where the entry includes: a first reference count and a first offset to an element in a first counter array, where the element contains a count of an action associated with the file identifier (Operation 610).In response to finding the entry, the system increments the first reference counter (Operation 612). The system releases the spinlock for the directory partition (Operation 614) and accesses the first counter array at the first offset to increment the action counter (Operation 616). The operation then returns to the previous state.

[0075] Fig. Figure 620 shows a flowchart illustrating a procedure that enables message transmission over a shared ring buffer queue using LFA-based shared memory access in a distributed system according to one aspect of the present application. The remote shared memory content comprises a global ring buffer queue; the allocated memory sections comprise segments of the global ring buffer queue; and each segment comprises: a queue in which data, including messages, is stored; an input semaphore and an output semaphore to synchronize access to the queue; and an input pointer and an output pointer to track positions relating to insert and remove operations performed on the queue (Operation 622).The system accesses the relevant segment of the global ring buffer queue containing the remote shared memory content through a first client node (Operation 624). The system obtains the input semaphore by decrementing it (Operation 626). In response to successfully obtaining the input semaphore, the system increments the input pointer (Operation 628). The system sends data via RDMA to a memory location pointed to by the input pointer (Operation 630). The system releases the output semaphore by incrementing it (Operation 632), and the operation is labeled C. Fig. continued.

[0076] Fig. Figure 640 shows a flowchart illustrating a procedure that enables message transmission over a shared ring buffer queue using LFA-based shared memory access in a distributed system, as described in one aspect of the present application. The system accesses the respective segment through a first server node that has allocated the respective segment (Operation 642). The system determines that the output semaphore has a value greater than zero (Operation 644). The system acquires the output semaphore by decrementing it (Operation 646). The system retrieves the data stored at the memory location pointed to by the output pointer using RDMA (Operation 648). The system increments the output pointer (Operation 650) and releases the input semaphore by incrementing it (Operation 652). The operation then returns to the input semaphore. Exemplary computer system and device

[0077] Fig. Figure 700 shows an exemplary computer system 700 that enables the dynamic allocation of memory in a distributed file system in accordance with one aspect of the present application. The computer system 700 comprises a processor 702, volatile memory 706, and a storage device 708. In some aspects, the computer system 700 may include a controller 704 (indicated by the dashed lines). The volatile memory 706 may, for example, comprise random-access memory (RAM) that serves as managed memory and can be used to store one or more memory pools. The storage device 708 may contain persistent memory that can be managed or accessed by the processor 702 (or the controller 704). Furthermore, the computer system 700 may be coupled with peripheral input / output user devices 710, such as a display device 711, a keyboard 712, and a pointing device 714.The storage device 708 can store an operating system 716, a content processing system 718 and data 736.

[0078] The content processing system 718 may contain instructions which, when executed by the computer system 700, can cause the computer system 700 or the processor 702 to perform the procedures and / or processes described in this disclosure. In particular, the content processing system 718 may contain instructions for receiving and sending data packets associated with an LFA API call (communication module 720).

[0079] The content processing system 718 can further include instructions for allocating a multitude of memory sections in a distributed system with a multitude of nodes, encompassing shared remote memory content (memory section allocation module 722). The content processing system 718 can include instructions for registering the allocated sections with an operating system so that they can be accessed via Remote Memory Access (RDMA) (memory section allocation module 722). The content processing system 718 can include instructions for a first node to access the allocated sections to obtain a local copy of the shared remote memory content (section terminal module 724).The content processing system 718 can also include instructions to perform an atomic operation on one or more bits of the shared remote memory content via libfabric-atomic (LFA) Application Programming Interface (API) calls (LFA API call module 726).

[0080] The content processing system 718 can additionally include instructions for updating one or more bits of the shared remote memory content based on a new value and an offset (LFA API call module 726 and bitmap management module 730). The content processing system 718 can also include instructions for retrieving a current value of one or more bits from the shared remote memory content based on the offset prior to the update (LFA API call module 726 and bitmap management module 730). Finally, the content processing system 718 can include instructions for performing an action on the shared remote memory content based on a comparison of the retrieved current value with an expected value in the local copy (LFA API call module 726 and bitmap management module 730).

[0081] The content processing system 718 can also contain instructions for accessing a table of contents for files and one or more counter arrays, as described above in relation to Fig. described (TOC Management Module 732). The content processing system 718 can contain instructions for accessing a queue (e.g., a message queue) in a segment of a global ring buffer queue, as described above in relation to the Fig. described (Queue Management Module 734).

[0082] The data 736 can include all data required as input or produced as output by the methods and / or processes described in this disclosure. In particular, the data 736 can store at least the following: a virtual copy of a global data structure; a local copy of a global data structure; an indicator of a segment, partition, blob, or portion of allocated memory; an indicator or identifier of a node, server node, or client node; an LFA API call; an atomic operation; an arithmetic operation; a logical operation; a value; an actual value, an expected value, or a new value; one or more bits; a bitmap data structure; an entry; a table of contents; a file identifier; a reference count; an offset; a spinlock; a global ring buffer queue; a queue; a message queue; data; atomic data;Encoded or decoded data; laminated data; a state; a word; a shared object; message data; a value for an input pointer, an output pointer, an input semaphore, or an output semaphore; an indicator for a file system or other distributed system; an indicator for a FAM module or other NVMe device; an extent bitmap and an extent map.

[0083] Fig. Figure 800 shows an exemplary device that enables the dynamic allocation of storage space in a distributed file system according to one aspect of the present application. Device 800 can comprise a plurality of units or devices that can communicate with each other via a wired, wireless, quantum light, or electrical communication channel. Device 800 can be implemented using one or more integrated circuits and can comprise fewer or more units or devices than those shown in Figure 800. Fig. The device shown includes [the device shown]. In addition, the device 800 can be integrated into a computer system or implemented as a separate device or devices that can communicate with other computer systems and / or devices.

[0084] Device 800 may also include a non-volatile memory system or a memory management unit. Device 800 may include modules or units 802-816 configured to perform similar functions or operations to modules 720-734 of Computer System 700. Fig. execute, including: a Communication Unit 802; a Memory Section Allocation Unit 804; a Section Terminal Unit 806; an LFA API Calling Unit 808; a Local Copy Management Unit 810; a Bitmap Management Unit 812; a TOC Management Unit 814; and a Queue Management Unit 816.

[0085] In general, the disclosed aspects provide a system that enables an LFA-based lockless, cluster-wide shared memory access API in a distributed system. In one aspect, the system allocates a multitude of memory sections within a distributed system comprising a multitude of nodes, each section containing shared remote memory. The system registers the allocated sections with an operating system for remote memory access (RDMA). The system accesses the allocated sections through a first node to obtain a local copy of the shared remote memory.The system performs an atomic operation on one or more bits of the shared remote memory content via libfabric-atomic (LFA) application programming interface (API) calls, which includes one or more of the following: updating the one or more bits of the shared remote memory content based on a new value and an offset; retrieving a current value of the one or more bits prior to updating from the shared remote memory content based on the offset; and performing an action on the shared remote memory content based on a comparison of the retrieved current value with an expected value in the local copy.

[0086] In one variation of this approach, the system performs the action on the shared remote memory content through the following operations. Upon determining that the current value does not match the expected value, the system receives an error message. Upon determining that the current value matches the expected value, the system performs the atomic operation on the one or more bits of the shared remote memory content, where the offset includes a global offset in the shared remote memory content associated with the one or more bits.

[0087] In another variation of this aspect, the system performs the atomic operation on the local copy by updating a shared object in the local copy based on the new value and offset, corresponding to one or more bits of the shared remote memory content.

[0088] In another variant, the atomic operation of one or more of the following includes: performing an arithmetic operation on the one or more bits, including one or more addition and subtraction operations; and performing a logical bitwise operation on the one or more bits, including one or more AND operations, OR operations, and exclusive OR (XOR) operations.

[0089] In another variant, the multitude of nodes comprises server nodes and client nodes, with the server nodes allocating the multitude of memory sections and the first node being either a client node or a server node. The client nodes perform the atomic operation by appending a shared object corresponding to one or more bits to an initial allocated portion based on the new value and the offset.

[0090] In another variation of this approach, the shared remote memory content is accessed via a file system associated with the distributed system, and this file system is a Fabric-Attached Memory File System (FAMfs). The server nodes are I / O nodes in the FAMfs. The allocation of the multiple memory sections is handled by allocation modules connected to each I / O node in the FAMfs, and the shared remote memory content comprises a bitmap data structure that tracks a state—used or free—of physical dimensions within the FAMfs's memory.

[0091] In another variant, the shared remote memory content comprises a table of contents for files and one or more counter arrays, and the allocated memory sections comprise partitions of the table of contents and the one or more counter arrays. The entries corresponding to the files in the table of contents include: a reference counter value indicating the number of processes currently having a corresponding file open, and an offset to an element in one of the counter arrays.

[0092] In another variation of this aspect, the system receives a request to open a file with a corresponding file identifier in the distributed system. The system determines a partition of the directory containing the file identifier. The system acquires a spinlock on the directory partition. The system searches the directory partition to find an entry corresponding to the file identifier. The entry includes a first reference count and a first offset to an element in a first counter array, where the element contains a count of an action associated with the file identifier. In response to finding the entry, the system increments the first reference count. The system releases the spinlock on the directory partition and accesses the first counter array at the first offset to increment the action's counter.

[0093] In another variation of this aspect, the remote shared memory content comprises a global ring buffer queue, and the allocated memory sections comprise segments of the global ring buffer queue. Each segment includes: a queue in which data, including messages, is stored; an input and an output semaphore to synchronize access to the queue; and an input pointer and an output pointer to track positions related to insert and remove operations performed on the queue.

[0094] In another variant, the system accesses the respective segment via the first client node by performing the following operations. The system obtains the input semaphore by decrementing it. In response to successfully obtaining the input semaphore, the system increments the input pointer. The system sends data via RDMA to a memory location pointed to by the input pointer, and the system releases the output semaphore by incrementing it.

[0095] In another variant, the system accesses the respective segment via a first server node that has assigned it, by performing the following operations. The system determines that the output semaphore has a value greater than zero. The system obtains the output semaphore by decrementing it and retrieves the data stored at the memory location pointed to by the output pointer using RDMA. The system increments the output pointer and releases the input semaphore by incrementing the input semaphore.

[0096] The data structures and code described in this detailed description are typically stored on a computer-readable storage medium, which can be any device or medium capable of storing code and / or data for use by a computer system. Computer-readable storage media include, but are not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tapes, CDs (Compact Discs), DVDs (Digital Versatile Discs or Digital Video Discs), or other media capable of storing computer-readable media known today or developed in the future.

[0097] The procedures and processes described in the "Detailed Description" section can be embodied as code and / or data, which can be stored on a computer-readable storage medium as described above. When a computer system reads and executes the code and / or data stored on the computer-readable storage medium, the computer system executes the procedures and processes that are embodied as data structures and code and stored on the computer-readable storage medium.

[0098] Furthermore, the procedures and processes described above can be integrated into hardware devices or apparatuses. These hardware devices or apparatuses can include, but are not limited to, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), dedicated or shared processors that execute a specific software program or piece of code at a specific time, and other known or subsequently developed programmable logic devices. When the hardware devices or apparatuses are activated, the hardware modules execute the procedures and processes they contain.

[0099] The foregoing descriptions of aspects serve only for illustration and description. They do not claim to be exhaustive and do not limit the aspects described herein to the forms shown. Accordingly, many modifications and variations will be obvious to those skilled in the art. Furthermore, the above disclosure is not intended to limit the aspects described herein. The scope of the aspects described herein is defined by the attached claims.

Claims

[1] A computer-implemented method comprising: Allocating a multitude of memory sections (130, 140, 150) comprising shared remote memory content in a distributed system comprising a multitude of nodes (110, 120, 138, 148, 158); Registering the allocated sections with an operating system for access via remote storage direct access (RDMA); Accessing the allocated sections by a first node (110) to obtain a local copy (111) of the shared remote memory content; and Performing an atomic operation on one or more bits of the shared remote memory content via libfabric-atomic-, LFA,-Application-Programming-Interface-, API, calls, including: Updating one or more bits of the shared remote memory content based on a new value and an offset; Retrieving a current value of one or more bits from the shared remote memory content based on the pre-update offset; and Performing an action on the shared remote memory content based on a comparison of the retrieved current value with an expected value in the local copy; the multitude of nodes includes server nodes (138, 148, 158) and client nodes (110, 120), where the server nodes allocate the multitude of memory sections, where the first node is either a client node or a server node, where the client nodes perform the atomic operation by connecting to a shared object corresponding to the one or more bits in a process based on the new value and Perform the first assigned section based on the offset, where the shared remote storage content is accessed by a file system connected to the distributed system, where the file system is a Fabric Attached Memory Filesystem, FAMfs, where the server nodes are I / O nodes in the FAMfs, where the allocation of the multitude of memory sections is performed by allocation modules connected to each I / O node in the FAMfs, and wherein the shared remote memory content comprises a bitmap data structure that tracks a status, used or free, of physical extents in the FAMf's memory. [2] The method of claim 1, wherein performing the action on the shared remote memory content comprises: In response to the finding that the current value does not match the expected value, an error message is received; and where the offset comprises a global offset in the shared remote memory content associated with the one or more bits. [3] The method according to claim 1, further comprising: Performing the atomic operation on the local copy by updating, in the local copy based on the new value and offset of a shared object, that corresponds to one or more bits of the shared remote memory content. [4] The method according to claim 1, wherein the shared remote memory content includes a file directory (320) and one or more counter arrays (340), wherein the allocated memory sections comprise partitions of the directory and of one or more counter arrays, and where the files in the table of contents include corresponding entries: a reference number (326) indicating the number of processes that currently have a corresponding file open; and an offset (328) to an element in one of the counter arrays. [5] The method according to claim 4, further comprising: Receiving a request to open a file with an associated file identifier in the distributed system; Determine a partition of the directory that contains the file identifier; Obtain (350) a spinlock (322) for the partition of the table of contents; Search (352) the partition of the directory to obtain an entry matching the file identifier, wherein the entry includes: a first reference number and a first offset to an element in a first counter array, where the element comprises a number of an action associated with the file identifier; in response to finding the entry, incrementing the first reference number (354); Release (356) the spinlock for the partition of the table of contents; and Accessing the first counter array at the first offset to increment the number of actions (358). [6] The method according to claim 1, where the shared remote memory content comprises a global ring buffer queue, where the allocated memory sections comprise segments (412, 422, 432) of the global ring buffer queue, and where each segment comprises: a queue (433) in which data, including messages, can be stored; an input semaphore (462) and an output semaphore (464) to synchronize access to the queue; and an input pointer (460) and an output pointer (466) to track positions with respect to insert and remove operations performed in the queue. [7] The method according to claim 6, further comprising access by the first client node to the respective segment by: Obtaining the input semaphore by decrementing the input semaphore; as a reaction to successfully acquiring the input semaphore, incrementing the input pointer; Sending data via RDMA to a memory location pointed to by the input pointer; and Releasing the output semaphore by incrementing the output semaphore. [8] The method according to claim 6, further comprising access by a first server node assigning the respective segment to the respective segment by: Determine that the output semaphore has a value greater than zero; Obtaining the output semaphore by decrementing the output semaphore; Retrieving data via RDMA that is stored at the memory location pointed to by the output pointer; Incrementing the output pointer and Releasing the input semaphore by incrementing the input semaphore. [9] A computer system (700), comprising: a processor (702) and a memory (708) connected to the processor that stores instructions which, when executed by the processor, cause the processor to perform a procedure, the procedure comprising: Allocating a multitude of memory sections (130, 140, 150) comprising shared remote memory content in a distributed system comprising a multitude of nodes (110, 120, 138, 148, 158); Registering the allocated sections with an operating system for access via remote storage direct access (RDMA); Accessing the allocated sections by a first node (110) to obtain a local copy (111) of the shared remote memory content; and Performing an atomic operation on one or more bits of the shared remote memory content via libfabric-atomic-, LFA,-Application-Programming-Interface-, API, calls, including: Updating one or more bits of the shared remote memory content based on a new value and an offset; Retrieving a current value of one or more bits prior to updating from the shared remote memory content based on the offset; and performing an action on the shared remote memory content based on a comparison of the retrieved current value with an expected value in the local copy; the multitude of nodes includes server nodes (138, 148, 158) and client nodes (110, 120), where the server nodes allocate the multitude of memory sections, where the first node is either a client node or a server node, where the client nodes perform the atomic operation by connecting to a shared object corresponding to the one and / or multiple bits in a process based on the new value and Perform the first assigned section based on the offset, where the shared remote storage content is accessed by a file system connected to the distributed system, where the file system is a Fabric Attached Memory Filesystem, FAMfs, where the server nodes are I / O nodes in the FAMfs, where the allocation of the multitude of memory sections is carried out by allocation modules connected to each I / O node in the FAMfs, and wherein the shared remote memory content comprises a bitmap data structure that tracks a status, used or free, of physical extents in the FAMf's memory. [10] The computer system according to claim 9, wherein performing the action on the shared remote memory content comprises: In response to the finding that the current value does not match the expected value, an error message is received; and In response to the finding that the current value matches the expected value, performing the atomic operation on the one or more bits of the shared remote memory content, where the offset comprises a global offset in the shared remote memory content associated with the one or more bits. [11] The computer system according to claim 9, wherein the method further comprises: Performing the atomic operation on the local copy by updating, in the local copy based on the new value and offset of a shared object, that corresponds to one or more bits of the shared remote memory content. [12] The computer system according to claim 9, wherein the shared remote memory content includes a file directory (320) and one or more counter arrays (340), wherein the allocated memory sections comprise partitions of the directory and of one or more counter arrays, and where the files in the table of contents include corresponding entries: a reference number (326) indicating the number of processes that currently have a corresponding file open; and an offset (328) to an element in one of the counter arrays. [13] The computer system according to claim 12, wherein the method further comprises: Receiving a request to open a file with an associated file identifier in the distributed system; Determine a partition of the directory that contains the file identifier; Obtain (350) a spinlock (322) for the partition of the table of contents; Search (352) the partition of the directory to obtain an entry matching the file identifier, wherein the entry includes: a first reference number and a first offset to an element in a first counter array, where the element comprises a number of an action associated with the file identifier; in response to finding the entry, incrementing the first reference number (354); Release (356) the spinlock for the partition of the table of contents; and Accessing the first counter array at the first offset to increment the number of actions (358). [14] The computer system according to claim 9, where the shared remote memory content comprises a global ring buffer queue, where the allocated memory sections comprise segments (412, 422, 432) of the global ring buffer queue, and where each segment comprises: a queue (433) in which data, including messages, can be stored; an input semaphore (462) and an output semaphore (464) to synchronize access to the queue; and an input pointer (460) and an output pointer (466) to track positions with respect to insert and remove operations performed in the queue. [15] The computer system according to claim 14, wherein the method further comprises access by the first client node to the respective segment by: Obtaining the input semaphore by decrementing the input semaphore; as a reaction to successfully acquiring the input semaphore, incrementing the input pointer; Sending data via RDMA to a memory location pointed to by the input pointer; and Releasing the output semaphore by incrementing the output semaphore. [16] A non-transitory computer-readable storage medium (708) which stores instructions which, when executed by a computer, cause the computer to perform a procedure comprising: Allocating a multitude of memory sections (130, 140, 160) comprising shared remote memory content in a distributed system comprising a multitude of nodes (110, 120, 138, 148, 158); Registering the allocated sections with an operating system for access via remote storage direct access (RDMA); Accessing the allocated sections by a first node (110) to obtain a local copy (111) of the shared remote memory content; and Performing an atomic operation on one or more bits of the shared remote memory content via libfabric-atomic, LFA, Application Programming Interface, API calls, including: Updating one or more bits of the shared remote memory content based on a new value and an offset; Retrieving a current value of one or more bits prior to updating from the shared remote memory content based on the offset; and performing an action on the shared remote memory content based on a comparison of the retrieved current value with an expected value on the local copy; the multitude of nodes includes server nodes (138, 148, 158) and client nodes (110, 120), where the server nodes allocate the multitude of memory sections, where the first node is either a client node or a server node, where the client nodes perform the atomic operation by connecting to a shared object corresponding to the one or more bits in a process based on the new value and Perform the first assigned section based on the offset, where the shared remote storage content is accessed by a file system connected to the distributed system, where the file system is a Fabric Attached Memory Filesystem, FAMfs, where the server nodes are I / O nodes in the FAMfs, where the allocation of the multitude of memory sections is carried out by allocation modules connected to each I / O node in the FAMfs, and wherein the shared remote memory content comprises a bitmap data structure that tracks a status, used or free, of physical extents in the FAMf's memory. [17] The non-transitory computer-readable storage medium according to claim 16, where the shared remote memory content comprises a global ring buffer queue, where the allocated memory sections comprise segments (412, 422, 432) of the global ring buffer queue, and where each segment comprises: a queue (433) in which data, including messages, can be stored; an input semaphore (462) and an output semaphore (464) to synchronize access to the queue; and an input pointer (460) and an output pointer (466) to track positions with respect to insert and remove operations performed in the queue. [18] The non-transitory computer-readable storage medium according to claim 17, further comprising access by the first client node to the respective segment by: Obtaining the input semaphore by decrementing the input semaphore; as a reaction to successfully acquiring the input semaphore, incrementing the input pointer; Sending data via RDMA to a memory location pointed to by the input pointer; and Releasing the output semaphore by incrementing the output semaphore. [19] The non-transitory computer-readable storage medium according to claim 17, further comprising accessing the respective segment by a first server node assigning the respective segment by: Determine that the output semaphore has a value greater than zero; Obtaining the output semaphore by decrementing the output semaphore; Retrieving data via RDMA that is stored at the memory location pointed to by the output pointer; Incrementing the output pointer; and Releasing the input semaphore by incrementing the input semaphore.

Images