Memory registration for remote direct memory access

Abstract

The invention relates to memory registration for remote direct memory access. Unused memory registration in a remote memory system is actively identified and de-registered. A memory server maintains a list of resource tracking entries for its uncompleted memory registration. A memory server continuously monitors a list of its resource tracking entries. When the memory server finds an unused memory registration in its list, it cancels the corresponding shared memory area. The monitoring may be performed in the background of the memory server by, for example, a monitoring thread. Thus, unused memory registrations may be quickly released.

Description

[0001] Government rights

[0002] This invention was made with Government support under Contract No. H98230-15-D-0022 / 0003 awarded by the Maryland Procurement Office. The Government has certain rights in this invention. Background Art

[0003] Remote Direct Memory Access (RDMA) is a networking technology designed to enhance data transfer efficiency and reduce latency in computing environments. RDMA enables direct memory access between networked devices without involving the device's processor, allowing high-throughput and low-latency communication between devices. This approach simplifies data transfer by offloading the data movement task from the processor to a dedicated RDMA-enabled network adapter. By facilitating direct communication between the memories of networked devices, RDMA can accelerate data-intensive applications such as high-performance computing, cloud computing, and storage systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Aspects of the present disclosure are best understood from the following detailed description when read with the accompanying figures.

[0005] Figure 1 is a block diagram of a remote memory system according to some embodiments.

[0006] Figure 2 is a block diagram of a remote memory system according to some embodiments.

[0007] Figure 3 is a timing diagram of a memory registration method according to some embodiments.

[0008] Figure 4 is a block diagram of a memory registration monitoring method according to some embodiments.

[0009] Figure 5 is a timing diagram of a memory release method according to some embodiments.

[0010] Figure 6 is a block diagram of a memory registration method for a compute node according to some embodiments.

[0011] Figure 7 is a block diagram of a computing node according to some embodiments.

[0012] Figure 8 is a block diagram of a storage registration method for a storage server according to some embodiments.

[0013] Figure 9is a block diagram of a storage server according to some embodiments.

[0014] Corresponding numerals and symbols in the different figures generally refer to corresponding parts unless otherwise indicated.The figures are drawn to clearly illustrate the relevant aspects of the disclosure and are not necessarily drawn to scale. DETAILED DESCRIPTION

[0015] The present disclosure provides many different examples for implementing different features. Specific examples of components and arrangements are described below to simplify the present disclosure. Of course, these are merely examples and are not intended to be limiting.

[0016] High-performance computing (HPC) environments can utilize client-server architectures. Remote memory systems (or split memory systems) can be implemented in client-server HPC environments using globally accessible memory servers (within a network). Memory servers can provide remote memory that can be shared by compute nodes. Memory servers are directly attached to a high-speed network structure that links both compute nodes and memory servers. Applications can create shared memory areas in memory servers, and compute nodes can access these shared memory areas using RDMA.

[0017] In RDMA, memory regions are registered with the memory server's network interface card (NIC) before memory access is performed at the memory server. When an HPC environment includes a large number of compute nodes, the amount of memory registration in the HPC environment can be significant, particularly for HPC computations with large memory requirements. Excessive memory registration can impact the performance and scalability of remote memory systems. Additionally, some network architectures may limit the total amount of memory registrations supported, which can restrict the amount of memory accessible to a compute node.

[0018] This disclosure describes a technique for efficiently deregistering unused memory in a remote memory system. A memory server maintains a list of resource tracking entries for its outstanding memory registrations. The resource tracking entries are stored in a predefined memory area. When a compute node requests information about a memory registration, the memory server provides the compute node with a key for the memory registration and the location of the resource tracking entry. The compute node can use the key to access a shared memory area at the memory server via Remote Direct Memory Access (RDMA) operations.

[0019] The resource tracking entry for a memory registration includes the state of the memory registration and the global reference count of the memory registration. The global reference count indicates the number of compute nodes (within the network) that are actively using the memory registration. The state can indicate whether the memory registration is active, ready for release, or inactive.

[0020] When a compute node is ready to release a memory registration, it directly updates the corresponding resource tracking entry on the memory server via (multiple) RDMA operations. For example, the compute node can perform an atomic compare-and-swap operation at the location of the resource tracking entry on the memory server. Thus, the compute node directly and atomically updates the global reference count and state of the memory registration. Advantageously, this allows the global reference count and state of the memory registration to be synchronized across compute nodes without degrading memory server performance or causing concurrent access challenges. For example, updating the resource tracking entry directly and atomically (from the compute node) can be faster and more reliable than updating the resource tracking entry through remote operations at the memory server.

[0021] The memory server continuously monitors the list of resource tracking entries for its memory registrations. Specifically, the memory server searches the list of unused memory registrations based on their global reference counts and / or states. When the memory server finds an unused memory registration in its list, it deregisters the corresponding shared memory area. Monitoring can be performed in the background of the memory server, for example, by a monitoring thread. Consequently, unused memory registrations can be quickly released, thereby reducing the number of memory registrations in the remote memory system. By quickly reducing the number of memory registrations, the performance of the remote memory system can be improved.

[0022] Figure 1 FIG1 is a block diagram of a remote memory system 100 according to some embodiments. The remote memory system 100 can be part of a computing environment, such as an HPC environment, that utilizes a client-server architecture. The remote memory system 100 includes a plurality of computing nodes 102 (including computing nodes 102A, 102B, and 102C), a network fabric 104, and one or more memory servers 106 (including memory servers 106A and 106B).

[0023] Computing nodes 102 work together to perform HPC calculations. For example, a task can be divided into smaller segments that can be parallelized across computing nodes 102. (Multiple) applications can be run on computing nodes 102 to perform HPC calculations. Computing nodes 102 can be independent units equipped with processors, memory, etc. (described later).

[0024] The network fabric 104 facilitates coordination and synchronization of the compute nodes 102 when performing HPC computations. The network fabric 104 may include routers, switches, links, and the like. The components of the network fabric 104 work together to provide a high-bandwidth interconnect between the compute nodes 102 and the storage servers 106. The design of the network fabric 104 may prioritize low latency and high throughput between the connected components. For example, the network fabric 104 may be based on technologies such as InfiniBand and Slingshot.

[0025] Memory server 106 provides remote memory that can be shared by compute nodes 102. HPC computations may have large memory requirements that may exceed the memory capacity of compute nodes 102. Compute nodes 102 can access the shared memory of memory server 106 and use the shared memory to perform HPC computations. Memory server 106 can be part of a remote memory system, such as fabric-attached memory (FAM), for compute nodes 102. An example implementation of FAM is OpenFAM. RDMA can be used by compute nodes 102 to access the shared memory of memory server 106 in a manner that meets the high throughput and low latency requirements of the computing environment.

[0026] Figure 2 is a block diagram of a remote memory system 200 according to some embodiments. The remote memory system 200 is a block diagram of a remote memory system 200 previously described with respect to FIG. Figure 1 An example implementation of remote memory system 100 is described. Additional details of remote memory system 200 are shown, including details of compute nodes 202 (including compute nodes 202A and 202B) and memory servers 206. For clarity of illustration, only one memory server 206 is shown.

[0027] Each computing node 202 includes various hardware components. These hardware components may include a processor 212, a memory 214, and a network interface card (NIC) 216. The hardware components may be interconnected via multiple buses and / or network connections. In one example, the processor 212, the memory 214, and the NIC 216 may be communicatively coupled via a bus 218 (such as a PCI-Express bus).

[0028] The processor 212 retrieves the executable code from the memory 214 and executes the executable code. When executed by the processor 212, the executable code may cause the processor 212 to implement any of the functions described herein. The processor 212 may be a microprocessor, an application specific integrated circuit, a microcontroller, or the like.

[0029] The memory 214 may include various types of memory, including volatile memory and non-volatile memory. For example, the memory 214 may include random access memory (RAM), read-only memory (ROM), hard disk drive (HDD), etc. Different types of memory may be used for different data storage needs. For example, the processor 212 may boot from the ROM, maintain non-volatile storage in the HDD, execute program code stored in the RAM, and store the data being processed in the RAM. The memory 214 may include a non-transitory computer-readable medium that stores instructions for execution by the processor 212. One or more modules within the computing node 202 may be partially or completely embodied as software and / or hardware for performing any of the functions described herein.

[0030] NIC 216 can be used to connect to a network (e.g., a LAN, WAN, etc.) and communicate with other devices over the network. NIC 216 facilitates the sending and receiving of data packets between the computing node 202 and the network and can comply with one or more networking standards, such as Ethernet, Wi-Fi, etc. In addition, NIC 216 supports RDMA, which allows direct memory access from one computing node 202 to another host (e.g., computing node 202 or memory server 206) without involving the processor 212, which can reduce latency and processing overhead. NIC 216 can utilize one or more RDMA-capable protocols (such as iWARP or RDMA over Converged Ethernet (RoCE)) to facilitate high-speed and low-latency data transfer between hosts. By transferring data access tasks from the processor 212 to the NIC 216, the network performance of the computing node 202 can be improved, which may be particularly advantageous for HPC applications.

[0031] The memory server 206 includes various hardware components. These hardware components may include a processor 222, a memory 224, and a NIC 226. The hardware components may be interconnected via multiple buses and / or network connections. In one example, the processor 222, the memory 224, and the NIC 226 may be communicatively coupled via a bus 228 (such as a PCI-Express bus).

[0032] The processor 222 retrieves the executable code from the memory 224 and executes the executable code. When the executable code is executed by the processor 222, the processor 222 can implement any of the functions described herein. The processor 222 can be a microprocessor, an application specific integrated circuit, a microcontroller, etc.

[0033] The memory 224 may include various types of memory, including volatile memory and non-volatile memory. For example, the memory 224 may include RAM, ROM, HDD, etc. Different types of memory may be used for different data storage needs. For example, the processor 222 may boot from ROM, maintain non-volatile storage in the HDD, execute program code stored in RAM, and store the data being processed in RAM. The memory 224 may include non-transitory computer-readable media that stores instructions for execution by the processor 222. One or more modules within the memory server 206 may be partially or completely embodied as software and / or hardware for performing any of the functions described herein.

[0034] NIC 226 can be used to connect to a network (e.g., a LAN, WAN, etc.) and communicate with other devices over the network. NIC 226 facilitates the sending and receiving of data packets between the memory server 206 and the network and can comply with one or more networking standards, such as Ethernet, Wi-Fi, etc. In addition, NIC 226 supports RDMA, which allows direct memory access from the memory server 206 to another host (e.g., a compute node 202 or a memory server 206) without involving the processor 222, which can reduce latency and processing overhead. NIC 226 can utilize one or more RDMA-capable protocols (such as iWARP or RoCE) to facilitate high-speed and low-latency data transfer between hosts. By transferring data access tasks from the processor 222 to the NIC 226, the network performance of the memory server 206 can be improved, which may be particularly advantageous for HPC applications.

[0035] When using RDMA, processes running on a host can bypass portions of the host's operating system network stack. For example, when a compute node 202 sends data to a memory server 206 via RDMA, the data to be sent may initially reside in the compute node 202's memory 214. Specifically, the data resides in the user space of the memory 214. The data can be copied directly from the user space of the memory 214 to the buffer of the NIC 216 of the compute node 202, bypassing the operating system kernel of the compute node 202. The data can then be sent to the NIC 226 of the memory server 206 via the network fabric. The data can then be copied directly from the buffer of the NIC 226 to the user space of the memory 224 of the memory server 206, bypassing the operating system kernel of the memory server 206. Similarly, when the memory server 206 sends data to the compute node 202 via RDMA, the aforementioned operations can be performed in reverse. By using RDMA, the copying of data from the user space of the memory to the operating system kernel (by the processor) can be avoided, thereby increasing the speed of data transfer and reducing the latency of data transfer.

[0036] In RDMA, before memory access, a shared memory region 224S in the memory 224 of the memory server 206 is registered with the NIC 226 of the memory server 206. The shared memory region 224S can be a region of volatile memory or a region of non-volatile memory. The shared memory region 224S will be accessed by the computing node 202 using RDMA operations. Registration of the shared memory region 224S may be referred to herein as "memory registration."

[0037] Depending on the architecture of the remote memory system 200, there may be constraints on the number of memory registers. For example, if the number of memory registers is large, the performance of the remote memory system 200 may be reduced because managing the memory registers may consume other network structures or system resources (such as open region files, memory-mapped I / O resources, internal cache, etc.). Additionally or alternatively, some remote memory architectures have limits on the total number of memory registers supported at a given time. To reduce the number of memory registers in the remote memory system 200, the memory server 206 proactively identifies unused memory registers and unregisters the corresponding shared memory regions 224S from the NIC 226.

[0038] The memory server 206 maintains a list of resource tracking entries 230 for outstanding memory registrations of its shared memory region 224S. The resource tracking entries 230 are stored in a predefined area of the memory 224, specifically in the tracking memory region 224T. The tracking memory region 224T can perform RDMA registration with the NIC 226, but the tracking memory region 224T can be different from the shared memory region 224S. For example, the tracking memory region 224T can be an optimized memory region that has a faster access speed than a non-optimized memory region (e.g., the shared memory region 224S).

[0039] Each resource tracking entry 230 corresponds to an RDMA registration made by a shared memory region 224S to a NIC 226. A resource tracking entry 230 includes the state of its memory registration (of the shared memory region 224S) and the global reference count of the memory registration (of the shared memory region 224S). The global reference count indicates the number of compute nodes 202 that are actively using the memory registration. The state indicates whether the memory registration is active, ready for release, or inactive. An active state indicates that the memory registration is valid. A ready for release state indicates that the memory registration is not being used by any compute node 202 and, therefore, the corresponding memory region 224S can be deregistered. An inactive state indicates that the corresponding memory region 224S has been deregistered.

[0040] In some embodiments, the resource tracking entry 230 of the memory register includes a binary word that includes a global reference count and a state. For example, the resource tracking entry 230 can be a 64-bit word. Some bits of the binary word can indicate a global reference count, wherein other bits of the binary word can indicate a state. That is, the first portion of the binary word can indicate the state of the memory register, while the second portion of the binary word can indicate the global reference count of the memory register. When the resource tracking entry 230 includes a binary word (which includes both a global reference count and a state), the global reference count and the state can be updated together by updating the binary word using an atomic operation (described later).

[0041] The memory server 206 continuously monitors the resource tracking entries 230. A process of the memory server 206 monitors the list of resource tracking entries 230 for unused memory registrations. An unused memory registration can be a memory registration whose global reference count of the corresponding resource tracking entry 230 is zero and / or whose status is ready for release, indicating that the corresponding shared memory region 224S is not used by any of the compute nodes 202. When the memory server 206 finds an unused memory registration in the list of resource tracking entries 230, it deregisters the corresponding shared memory region 224S from the NIC 226. The monitoring and deregistration can be performed in the background of the memory server 206 (e.g., by a monitoring thread), which can help reduce the latency observed by the compute node 202 when the shared memory region 224S is being deregistered. Therefore, the amount of memory registrations can be reduced without affecting the performance of the existing workload on the compute node 202.

[0042] When a shared memory area 224S is deregistered, the state of the corresponding resource tracking entry 230 is changed to inactive. In addition, the memory allocation for the resource tracking entry 230 can be released (e.g., within the tracking memory area 224T). In some embodiments, the resource tracking entry 230 is moved to a garbage collector queue. A process of the memory server 206 can monitor the garbage collector queue and can de-allocate the memory for the resource tracking entry 230 in the garbage collector queue. The de-allocation can be performed in the background of the memory server 206 (e.g., by a garbage collector thread). The process used to perform garbage collection can be different from the process used to monitor unused memory registrations.

[0043] When a process on a compute node 202 desires to access a shared memory region 224S on a memory server 206, the compute node 202 requests the memory registration information for the shared memory region 224S from the memory server 206. The shared memory region 224S is registered (if it has not already been registered), and a corresponding resource tracking entry 230 may be created or updated (such as by incrementing its global reference count and / or changing its state). The memory registration information is then sent back to the compute node 202. The memory registration information includes a key for the memory registration of the shared memory region 224S. The compute node 202 can remotely access the shared memory region 224S by performing an RDMA operation (e.g., a read (get) or a write (put)) using the key. In addition, the memory registration information includes the location of the corresponding resource tracking entry 230 in the memory registration. The location of the resource tracking entry 230 may be the location (e.g., offset) within the tracking memory region 224T where the resource tracking entry 230 is stored. When computing node 202 no longer expects to access shared memory area 224S, it releases shared memory area 224S by remotely updating the corresponding resource tracking entry 230. The global reference count of resource tracking entry 230 can be decremented and / or the state of resource tracking entry 230 can be changed. Computing node 202 directly writes the new values of these fields of resource tracking entry 230 to the location specified by the memory registration information (in tracking memory area 224T). Computing node 202 can directly update resource tracking entry 230 by performing an atomic compare and exchange operation at the location specified by the memory registration information (in tracking memory area 224T). When resource tracking entry 230 includes a binary word, computing node 202 can write the new binary word (with the updated global reference count and / or state) to the location in tracking memory area 224T. Therefore, the binary word is overwritten.

[0044] It may be advantageous for a compute node 202 to directly update a resource tracking entry 230 within a tracking memory area 224T of a memory server 206. The resource tracking entry 230 may be kept synchronized on the remote memory system 200. When many compute nodes 202 are accessing a memory server 206, updating the resource tracking entry 230 directly from a compute node 202 may avoid concurrent access challenges that may arise from other update techniques, such as remote operations at the memory server 206.

[0045] In some embodiments, the memory registration information is cached in a local resource cache of the compute node 202. The local resource cache can be stored in the memory 214 of the compute node 202 and can be shared by processes running on the compute node 202. When a client process on the compute node 202 desires to access the shared memory area 224S of the memory server 206, an agent can first search the local resource cache for the memory registration information. The agent can be part of the client process accessing the shared memory area 224S, or part of a management process used to manage the local resource cache. If the memory registration information is available in the local resource cache, the memory registration information can be directly obtained from the local resource cache and used by the client process. If the memory registration information is not available in the local resource cache, the agent can request the memory registration information from the memory server 206 (as described above). Caching the memory registration information on the compute node 202 can further optimize the overall performance of the remote memory system 200, such as by avoiding request storms at the memory server 206.

[0046] The compute node 202 can maintain a local reference count 232 for each outstanding memory registration it uses at the memory server 206. The local reference count 232 is a count of how many client processes of the compute node 202 are currently using the memory registration. When a client process of the compute node 202 begins using a shared memory region 224S, the corresponding local reference count 232 for that memory registration is incremented. When a client process of the compute node 202 stops using a shared memory region 224S, the corresponding local reference count 232 for that memory registration is decremented. When the corresponding local reference count 232 reaches zero, the compute node 202 can release the shared memory region 224S (as described above).

[0047] Figure 3 3 is a timing diagram of a memory sharing method 300 according to some embodiments. The memory sharing method 300 may be executed in a remote memory system when a compute node requests memory registration information for a shared memory region of a memory server. When a client process of a compute node desires to access a shared memory region via RDMA, the compute node may request the memory registration information.

[0048] In step 302, the computing node checks its local resource cache for the memory registration information. If the memory registration information is available in the local resource cache, the computing node retrieves the memory registration information and returns it to the client process. Additionally, the local reference count for the memory registration information is incremented. At this point, the memory sharing method 300 terminates. If the memory registration information is not available in the local resource cache, steps 304 through 314 are executed to request the memory registration information from the memory server.

[0049] In step 304, the computing node sends a request for memory registration information to the memory server. The request may invoke a remote operation at the memory server. For example, the request may be a remote procedure call (RPC) request. The request may include, for example, an identifier of the desired memory registration.

[0050] The storage server checks its list of resource tracking entries for a resource tracking entry corresponding to the storage registration identified by the request from the compute node. If a corresponding resource tracking entry is found in the list, step 306 is executed. If the list does not contain a corresponding resource tracking entry, step 308 is executed.

[0051] In step 306 (if the desired resource tracking entry exists), the memory server updates the resource tracking entry corresponding to the requested memory registration. The resource tracking entry is updated based on the state of the resource tracking entry. If the state of the resource tracking entry is active, the global reference count of the resource tracking entry is incremented by (one). If the state of the resource tracking entry is ready to release, the state of the resource tracking entry is changed to active, and the global reference count of the resource tracking entry is incremented by (one). Atomic operations can be used to update the resource tracking entry, particularly when the resource tracking entry includes a binary word containing two fields of the resource tracking entry.

[0052] If the state of the resource tracking entry is inactive, the resource tracking entry is considered invalid and is replaced with a new resource tracking entry. The shared memory region is registered with the NIC of the memory server, a new resource tracking entry for the memory registration is created, and the new resource tracking entry is added to the list of resource tracking entries. The state of the new resource tracking entry is set to active, and the global reference count of the new resource tracking entry is set to one. The memory allocation for the invalid resource tracking entry is deallocated. For example, the invalid resource tracking entry can be moved to a garbage collector queue, which is monitored and processed by another process of the memory server.

[0053] In step 308 (if the desired resource tracking entry does not exist), the memory server registers the shared memory area with its NIC. A new resource tracking entry for the memory registration is created and added to the list of resource tracking entries. The state of the new resource tracking entry is set to active, and the global reference count of the new resource tracking entry is set to one. Memory is allocated for the new resource tracking entry from a predefined memory area for the resource tracking entry (e.g., the memory server's tracking memory area).

[0054] In step 310, the memory server sends memory registration information to the compute node. The memory registration information may be created by the memory server. The memory registration information includes a key for the memory registration, which the compute node may use to write to the shared memory area. The memory registration information further includes the location of the resource tracking entry for the memory registration. The location may be an offset within a predefined memory area of the resource tracking entry.

[0055] In step 312, the computing node stores the memory registration information in its local resource cache so that the memory registration information can be used by other processes of the computing node in subsequent requests.

[0056] In step 314, the computing node increments the local reference count of the memory registration by one. The local reference count may also be stored in a local resource cache of the computing node.

[0057] The above steps of the computing node may be performed by a client process that desires to access the shared memory area of the memory server, or may be performed by a management process of the computing node (which is different from the client process). For example, the management process may check the local resource cache, request the memory registration information, and receive the memory registration information. In this case, the management process may pass the memory registration information to the requesting client process so that the client process can use the information to access the shared memory area.

[0058] Figure 4 4 is a block diagram of a memory registration monitoring method 400 according to some embodiments. The memory registration monitoring method 400 may be executed by a process of a memory server. In some embodiments, the memory registration monitoring method 400 may be executed in the background of the memory server (eg, by a monitoring thread).

[0059] The memory server may perform step 402 of checking its list of resource tracking entries for resource tracking entries of unused memory registrations. If a resource tracking entry in the list has a status of ready to be released, the resource tracking entry may be used for the unused memory registration. Additionally or alternatively, if the global reference count of a resource tracking entry in the list is zero, the resource tracking entry may be used for the unused memory registration. Such a memory registration is a candidate for deregistration.

[0060] The memory server may perform step 404 of deregistering the shared memory region (from the memory server's NIC). The deregistered shared memory region corresponds to the unused memory registration identified in step 402. The resource tracking entry for the memory registration may be updated, specifically by changing its state to inactive. Memory allocation for the resource tracking entry may then be released. For example, the resource tracking entry may be removed from a list of resource tracking entries and moved to a garbage collector queue.

[0061] Figure 5 is a timing diagram of a memory release method 500 according to some embodiments. The memory release method 500 may be executed in a remote memory system when a compute node releases memory registration for a shared memory area of a memory server. The steps described subsequently for a compute node may be executed by a client process desiring to release a shared memory area of a memory server, or may be executed by a management process of the compute node.

[0062] In step 502, the compute node decrements the local reference count of the memory register by one. The local reference count can be stored in the compute node's local resource cache. As previously described, the local reference count is a count of how many client processes on the compute node are currently using the memory register. If the local reference count is zero after decrement, steps 504 and 506 are executed.

[0063] In step 504, the computing node invalidates the memory registration information in its local resource cache. Therefore, the memory registration information may no longer be available to other processes of the computing node.

[0064] In step 506, the compute node directly updates the resource tracking entry of the memory registration at the memory server. The resource tracking entry is updated via (multiple) RDMA operations. For example, the compute node can perform an atomic compare-and-swap operation at the location of the resource tracking entry on the memory server. The location of the resource tracking entry can be a location (e.g., an offset) within the tracking memory region of the memory server. The location of the resource tracking entry is known to the compute node because it is part of the memory registration information that the compute node previously received during registration of the shared memory region.

[0065] A resource tracking entry can only be updated if its status is active. If the status of a resource tracking entry is inactive, the resource tracking entry cannot be updated. When a resource tracking entry is updated, the global reference count of the resource tracking entry may be decremented and / or the status of the resource tracking entry may be changed. If the status of the resource tracking entry is active, the global reference count is decremented by one. Additionally, if the global reference count is (or will be) decremented to zero, the status of the resource tracking entry is changed to ready for release.

[0066] Subsequently, the process of the memory server can deregister the shared memory region from the NIC of the memory server (as described above). Specifically, if the memory registration meets the above criteria of unused memory registration after updating the resource tracking entry, the shared memory region can be deregistered.

[0067] Some variations are contemplated. In some embodiments, other types of memory (including non-volatile memory) can be shared. For example, regions of a storage device (e.g., a solid-state device (SSD)) can be shared via RDMA. In some embodiments, memory registration information can be requested from a memory server using RDMA (rather than RPC). For example, a resource tracking entry can be directly updated by a client node to increment the global reference count of the resource tracking entry. In some embodiments, the health of a compute node is tracked by a memory server, such as by tracking processes running on the compute node that are using the memory registration of the memory server. For example, a compute node can periodically send health update messages to a memory server to notify the memory server that the compute node is healthy. If a memory server detects that a compute node has crashed (e.g., no health update messages have been received within a certain time frame), the memory server updates the resource tracking entries of any memory registrations used by the compute node. Specifically, the global reference count of any resource tracking entry used by the compute node can be decremented by one and / or the state of the resource tracking entry can be changed to ready for release (as described above).

[0068] Figure 6 is a block diagram of a memory registration method 600 according to some embodiments. The memory registration method 600 may be performed by a computing node in a remote memory system.

[0069] The computing node may perform step 602 of obtaining memory registration information from a memory server, the memory registration information including a key for a memory registration of a shared memory area of the memory server, the memory registration information further including a location of a resource tracking entry for the memory registration, the location of the resource tracking entry including a location within the tracking memory area of the memory server. The resource tracking entry may include a state of the memory registration and a global reference count of the memory registration. The resource tracking entry may include a binary word, a first portion of the binary word indicating the state of the memory registration, and a second portion of the binary word indicating the global reference count of the memory registration.

[0070] Obtaining the memory registration information may include: searching a local resource cache of the computing node for the memory registration information; requesting the memory registration information from a memory server in response to the memory registration information not being available in the local resource cache; and (optionally) obtaining the memory registration information from the local resource cache in response to the memory registration information being available in the local resource cache. Requesting the memory registration information from the memory server may include sending a remote procedure call request to the memory server.

[0071] The compute node may perform step 604 of accessing the shared memory area of the memory server by performing a remote direct memory access operation using the key.

[0072] The compute node may perform step 606 of updating the resource tracking entry of the memory register by directly writing to the location within the tracking memory area of the memory server. Writing to the location within the tracking memory area of the memory server may include performing an atomic compare and swap operation at the location within the tracking memory area of the memory server. When the resource tracking entry includes a binary word, updating the resource tracking entry of the memory register may include overwriting the binary word.

[0073] Additional steps (not separately illustrated) may be performed by the compute node. For example, the compute node may perform a step of decrementing a local reference count of a memory register at the compute node, wherein the resource tracking entry of the memory register is updated in response to the local reference count being zero.

[0074] Figure 7 is a block diagram of a computing node 700 according to some embodiments. The computing node 700 may include a processor 702 and a memory 704. The memory 704 may be a non-transitory computer-readable medium storing a program for execution by the processor 702. In this embodiment, one or more modules within the computing node 700 may be embodied in part or in whole as software for performing any of the functions described herein. For example, the memory 704 may include: instructions 706 for obtaining memory registration information from a memory server, the memory registration information including a key for a memory registration of a shared memory area of the memory server, the memory registration information further including a location of a resource tracking entry of the memory registration, the location of the resource tracking entry including a location within a tracking memory area of the memory server; instructions 708 for accessing the shared memory area of the memory server by performing a remote direct memory access operation using the key; and instructions 710 for updating a resource tracking entry of the memory registration by directly writing to a location within the tracking memory area of the memory server.

[0075] Figure 8is a block diagram of a memory registration method 800 according to some embodiments. The memory registration method 800 may be performed by a memory server in a remote memory system.

[0076] The memory server may perform step 802 of creating a memory registration by registering a shared memory area of the memory server with a network interface card of the memory server. The shared memory area may be an area of volatile memory. The shared memory area may be an area of non-volatile memory.

[0077] The memory server may perform step 804 of storing a resource tracking entry for the memory registration in a tracking memory area of the memory server. The resource tracking entry may include a state of the memory registration and a global reference count of the memory registration. The resource tracking entry may include a binary word, a first portion of the binary word indicating the state of the memory registration, and a second portion of the binary word indicating the global reference count of the memory registration.

[0078] The memory server may perform step 806 of sending memory registration information to the computing node, where the memory registration information includes a key for memory registration and a location of a resource tracking entry, where the location of the resource tracking entry includes a location within a tracking memory area of the memory server.

[0079] Additional steps (not separately shown) may be performed by the memory server. The memory server may deregister the shared memory region from the network interface card in response to the memory registration being in a ready-to-release state. After the shared memory region is deregistered, the memory server may de-allocate the memory for the resource tracking entry. The resource tracking entry may be moved to a garbage collector queue for de-allocation. Deregistering the shared memory region may be performed in the background of the memory server.

[0080] Figure 9is a block diagram of a memory server 900 according to some embodiments. The memory server 900 may include a processor 902 and a memory 904. The memory 904 may be a non-transitory computer-readable medium storing a program for execution by the processor 902. In this embodiment, one or more modules within the memory server 900 may be embodied in part or in whole as software for performing any of the functions described herein. For example, the memory 904 may include: instructions 906 for creating a memory registration by registering a shared memory region of the memory server with a network interface card of the memory server; instructions 908 for storing a resource tracking entry of the memory registration in a tracking memory region of the memory server; and instructions 910 for sending memory registration information to a compute node, the memory registration information including a key for the memory registration and a location of a resource tracking entry, the location of the resource tracking entry including a location within the tracking memory region of the memory server.

[0081] The foregoing has summarized the features of multiple examples so that those skilled in the art may better understand the various aspects of the present disclosure. Various modifications and combinations of the illustrative examples and other examples will be apparent to those skilled in the art after reference to the description. Therefore, the appended claims are intended to cover any such modifications.

Claims

1. A method performed by a computing node, the method comprising: obtaining storage registration information from a storage server, the storage registration information comprising a key for a storage registration of a shared memory area of the storage server, the storage registration information further comprising a location of a resource tracking entry of the storage registration, the location of the resource tracking entry comprising a location within a tracking memory area of the storage server; accessing the shared memory area of the memory server by performing a remote direct memory access operation using the key; as well as The resource tracking entry of the memory registry is updated by writing directly to the location within the tracking memory area of the memory server.

2. The method according to claim 1, wherein Writing to the location within the tracking memory area of the memory server includes performing an atomic compare-and-swap operation at the location within the tracking memory area of the memory server.

3. The method according to claim 1, wherein The resource tracking entry includes a state of the memory registration and a global reference count of the memory registration.

4. The method according to claim 3, wherein: The resource tracking entry includes a binary word, a first portion of the binary word indicating the state of the memory register, and a second portion of the binary word indicating the global reference count of the memory register.

5. The method according to claim 4, wherein Updating the resource tracking entry of the memory register includes overwriting the binary word.

6. The method according to claim 1, further comprising: A local reference count of the memory registration is decremented at the compute node, wherein the resource tracking entry of the memory registration is updated in response to the local reference count being zero.

7. The method according to claim 1, wherein Obtaining the memory registration information includes: searching for the memory registration information in a local resource cache of the computing node; and The memory registration information is requested from the memory server in response to the memory registration information being unavailable in the local resource cache.

8. The method according to claim 7, wherein: Obtaining the memory registration information further includes: The memory registration information is obtained from the local resource cache in response to the memory registration information being available in the local resource cache.

9. The method according to claim 7, wherein: Requesting the storage registration information from the storage server includes sending a remote procedure call request to the storage server.

10. A method performed by a storage server, the method comprising: creating a memory registry by registering a shared memory area of the memory server with a network interface card of the memory server; storing a resource tracking entry of the storage registration in a tracking storage area of the storage server; as well as Memory registration information is sent to a computing node, the memory registration information including a key for the memory registration and a location of the resource tracking entry, the location of the resource tracking entry including a location within the tracking memory area of the memory server.

11. The method according to claim 10, wherein: The resource tracking entry includes a state of the memory registration and a global reference count of the memory registration.

12. The method according to claim 11, wherein The resource tracking entry includes a binary word, a first portion of the binary word indicating the state of the memory register, and a second portion of the binary word indicating the global reference count of the memory register.

13. The method according to claim 11, further comprising: The shared memory area is deregistered from the network interface card in response to the state of the memory registration being ready to be released.

14. The method according to claim 13, further comprising: After the shared memory region is deregistered, memory for the resource tracking entry is de-allocated.

15. The method according to claim 13, wherein The deregistration of the shared memory area is performed in the background of the memory server.

16. The method according to claim 10, wherein The shared memory area is an area of volatile memory.

17. The method according to claim 10, wherein The shared memory area is an area of a nonvolatile memory.

18. A system comprising: A storage server, wherein the storage server is configured to: creating a memory registry of a shared memory area of the memory server; storing a resource tracking entry of the storage registration in a tracking storage area of the storage server; as well as creating storage registration information, the storage registration information including a key for the storage registration and a location of the resource tracking entry, the location of the resource tracking entry including a location within the tracking storage area of the storage server; as well as A computing node, wherein the computing node is configured to: obtaining the storage registration information from the storage server; accessing the shared memory area of the memory server by performing a remote direct memory access operation using the key; as well as The resource tracking entry of the memory registry is updated by writing directly to the location within the tracking memory area of the memory server.

19. The system according to claim 18, wherein: The resource tracking entry includes a binary word, a first portion of the binary word indicating a state of the memory register, and a second portion of the binary word indicating a global reference count of the memory register.

20. The system of claim 19, wherein: The memory server is further configured to: unregister the shared memory area after the computing node updates the resource tracking entry; and Moves the resource tracking entry to the garbage collector queue.

Images