REMOTE STORE DIRECT OPERATION (RDMO) FOR APPEND

Abstract

A system includes a first network device and a second network device. The first network device is configured to send a command over a network that specifies a value and instructs that the value be inserted into a set of values ​​in a memory. The second network device is configured to receive the command over the network and execute the command by inserting the value into the set of values.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is related to a U.S. patent application entitled "Maximum Compare-and-Swap Remote Direct Memory Operation (RDMO)," Attorney Docket Number 23-TV-1086US01; a U.S. patent application entitled "Hash Table Remote Direct Memory Operations (RDMO)," Attorney Docket Number 23-TV-1086US02; and a U.S. patent application entitled "Remote Logging Remote Direct Memory Operations (RDMO)," Attorney Docket Number 23-TV-1086US04, all filed on the same date. The disclosures of these related applications are incorporated herein by reference. FIELD OF THE INVENTION

[0002] The present invention relates generally to network communications and, more particularly, to transport protocol-based remote memory direct operations. BACKGROUND OF THE INVENTION

[0003] Remote Direct Memory Access (RDMA) is a transport protocol that enables network devices to transfer data to and from remote storage without host involvement. RDMA transport can occur over Infiniband™ or Ethernet networks, for example. SUMMARY OF THE INVENTION

[0004] The invention is defined by the claims. To illustrate the invention, aspects and embodiments are described herein, which may or may not fall within the scope of the claims.

[0005] An embodiment described herein provides a system comprising a first network device and a second network device. The first network device is configured to send a command over a network specifying a value and instructing that the value be inserted into a set of values ​​in a memory. The second network device is configured to receive the command over the network and execute the command by inserting the value into the set of values.

[0006] In one disclosed embodiment, in response to the command, the second network device is configured to read a pointer pointing to a memory location into which the value is to be inserted and to write the value to the memory location. In some embodiments, the command is embedded in a transport protocol used by the first and second network devices. In one embodiment, the transport protocol is a remote direct memory access (RDMA) protocol. In one embodiment, the second network device is configured to execute the command atomically.

[0007] Furthermore, according to an embodiment described herein, a network device is provided with a network interface and processing circuitry. The network interface is provided for connection to a network. The processing circuitry is provided for sending a command over the network that specifies a value and instructs that the value be inserted into a set of values ​​in a memory.

[0008] In some embodiments, the command is embedded in a transport protocol used by the network device. In one embodiment, the transport protocol is a remote direct memory access (RDMA) protocol.

[0009] Furthermore, according to an embodiment described herein, a network device is provided with a network interface and processing circuitry. The network interface is provided for connection to a network. The processing circuitry is provided for receiving a command over the network that specifies a value and instructs that the value be inserted into a set of values ​​in a memory, and for executing the command by inserting the value into the set of values.

[0010] In one disclosed embodiment, in response to the command, processing circuitry is provided for reading a pointer pointing to a memory location into which the value is to be inserted and writing the value to the memory location. In some embodiments, the command is embedded in a transport protocol used by the network device. In one embodiment, the transport protocol is a remote direct memory access (RDMA) protocol. In one embodiment, processing circuitry is provided for atomically executing the command.

[0011] Furthermore, according to an embodiment described herein, a method is provided that includes sending a command specifying a value and instructing that the value be inserted into a set of values ​​in a memory from a first network device over a network. The command is received over the network at a second network device. The command is executed at the second network device by inserting the value into the set of values.

[0012] Any feature of one aspect or embodiment may be applied to other aspects or embodiments, in any suitable combination. In particular, any feature of a method aspect or embodiment may be applied to a device aspect or embodiment, and vice versa.

[0013] The present invention will be better understood from the following detailed description of embodiments thereof taken in conjunction with the drawings. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a block diagram schematically illustrating a computing system employing remote direct memory operations (RDMO) in accordance with an embodiment of the present invention; Fig. 2 is a flowchart schematically illustrating a method for performing a maximum compare-and-swap (MAX-CAS) RDMO instruction according to an embodiment of the present invention; Fig. 3 is a flowchart schematically illustrating a method for performing a Hash Table Get RDMO instruction with respect to a hash table according to an embodiment of the present invention; Fig. 4 is a flowchart schematically illustrating a method for performing a Table Append RDMO instruction for inserting into a table according to an embodiment of the present invention; Fig. 5 is a block diagram schematically illustrating a computing system employing remote logging using RDMO according to an embodiment of the present invention; and Fig. 6 is a flowchart schematically illustrating a method for remote logging using RDMO according to an embodiment of the present invention. DETAILED DESCRIPTION OF EMBODIMENTS OVERVIEW

[0014] Embodiments of the present invention described herein provide improved methods and systems for performing complex operations directly on remote memory. The disclosed methods are referred to herein as "Remote Direct Memory Operations" (RDMO). In contrast to simple actions such as remote reads and writes, the disclosed RDMO instructions perform complex operations that may include multiple memory access operations, arbitration, table and pointer manipulation, and the like.

[0015] In a typical configuration, a computing system includes first and second network devices that communicate over a network. The first network device sends an RDMO command over the network to the second network device, and the second network device executes the command directly in memory. The network devices may include, for example, network interface controllers (NICs) or data processing units (DPUs, sometimes referred to as "smart NICs").

[0016] In one example, the RDMO instruction is a maximum compare-and-swap (MAX-CAS) instruction. The MAX-CAS instruction specifies a memory location, a compare value, and an exchange value, and instructs that the exchange value be written to the memory location only if the compare value is greater than a current value in the memory location. In another example, the RDMO instruction is a hash table get / set instruction that instructs the second network device to get or set a value in a hash table. Another example is a table append instruction that inserts a new value at the end of a table in memory. Another example involves RDMO instructions that perform fault-tolerant remote logging.

[0017] The disclosed RDMO instructions enable complex operations to be performed in remote memory with minimal latency (eliminating the need to wait for multiple round-trip transactions across the network) and without requiring the involvement of a remote host. In some embodiments, the disclosed RDMO instructions are fully embedded in the transport protocol used by the network devices. For example, the instructions may be implemented as extensions to the RDMA protocol.

[0018] When executing a specific RDMO command, the second network device typically performs the command's various operations atomically. Atomic execution of RDMO commands is important, for example, in distributed applications where multiple clients can access memory concurrently.

[0019] Alternative, naive solutions for performing a complex operation remotely could involve executing a sequence of conventional RDMA transactions or using remote procedure call (RPC) techniques to invoke remote procedures. Such approaches are proposed, for example, by Brock et al. in "RDMA vs. RPC for Implementing Distributed Data Structures," Proceedings of the 2019 IEEE / ACM 9th Workshop on Irregular Applications: Architectures and Algorithms (IA3), November 2019. However, these approaches are highly suboptimal because they incur significant latency and communication overhead and / or require support from a remote host. SYSTEM DESCRIPTION

[0020] Fig. Figure 1 is a block diagram schematically illustrating a computer system 20 employing remote memory direct operations (RDMO) according to one embodiment of the present invention. System 20 includes network devices 24A and 24B that support RDMO instructions. In the present example, network devices 24A and 24B are NICs. In general, however, the disclosed techniques may be implemented in any other suitable type of network device, such as DPUs ("Intelligent NICs"), network-capable graphics processing units (GPUs), etc.

[0021] Network device 24A (referred to as NIC1) serves a host 28A (referred to as HOST1), and network device 24B (referred to as NIC2) serves a host 28B (referred to as HOST2). NICs 24A and 24B communicate over a network 32. Network 32 may comprise, for example, an InfiniBand or Ethernet network. Each NIC communicates locally with its host over a peripheral bus 36, such as a Peripheral Component Interconnect Express (PCIe) or Nvlink bus. NIC2 also communicates locally with a memory 40 over bus 36. Memory 40 may be, for example, random-access memory (RAM) or flash memory.

[0022] In the following examples, network device 24A (NIC1) sends RDMO commands to network device 24B (NIC2) for execution in memory 40. NIC2 executes the RDMO commands directly in memory 40 without requiring the involvement of HOST2. In this context, network device 24A (NIC1) is also referred to as the "initiator NIC" and network device 24B (NIC2) is also referred to as the "target NIC." The roles of initiator and target are defined for a given RDMO command. In general, a given NIC can serve as the initiator for some RDMO commands and the target for other RDMO commands, possibly simultaneously.

[0023] As mentioned above, the disclosed RDMO commands are embedded in the transport protocol used by NIC1 and NIC2. In this example, the transport protocol is RDMA. Alternatively, RDMO commands can be embedded in any other suitable transport protocol.

[0024] In the example of Fig. 1, each NIC includes a host interface (I / F) 44 for communication over bus 36, a network I / F 48 for communication with network 32, and processing circuitry 52 that performs the various processing tasks of the NIC, including initiating and / or executing RDMO instructions.

[0025] The Fig. The configuration of system 20 shown in Figure 1 is a simplified configuration chosen solely for conceptual clarity. In alternative embodiments, any other suitable system configuration may be used. For example, system 20 may include a large number of hosts and NICs (or other network devices) that support RDMO. Performing complex operations in remote storage using RDMO commands

[0026] The following section describes several illustrative examples of RDMO commands that may be supported by System 20 NIC1 and NIC2. Maximum Compare-and-Swap (MAX-CAS)

[0027] In some embodiments, NIC1 and NIC2 support an RDMO instruction referred to as Maximum Compare-and-Swap (MAX-CAS). The MAX-CAS instruction specifies (i) a memory location in memory 40, (ii) a compare value, and (iii) a swap value. The instruction instructs the target network device to write the swap value to the memory location if (and only if) the compare value is greater than the current value found in the memory location. This is in contrast to the known RDMA-CAS instruction, which writes the swap value to the memory location if (and only if) the compare value is equal to the current value found in the memory location. The disclosed MAX-CAS instruction is useful, for example, to ensure that a certain value (e.g., a version number) is only incremented and never decremented.

[0028] Fig. Figure 2 is a flowchart schematically illustrating a method for executing a MAX-CAS RDMO command according to an embodiment of the present invention. The method begins with NIC1 (the initiator NIC) sending a MAX-CAS command over network 32 to NIC2 (the target NIC) at a command send stage 60. NIC2 receives the command over network 32 at a command receive stage 64.

[0029] At a readout stage 68, NIC2 reads the current value from the memory location specified in the command. At a comparison stage 72, NIC2 compares the current value of the memory location with the comparison value specified in the command. If the comparison value is not greater than the current value, NIC2 does not change the current value of the memory location, and the process terminates at an abort stage 80. If, however, the comparison value is greater than the current value, NIC2 writes the exchange value specified in the command to the memory location in a write stage 76 instead of the current value.

[0030] NIC2 typically executes stages 68, 72, and 76 atomically, meaning it does not allow any intermediate operations at the respective memory location. Operation atomicity is important, for example, if memory 40 is accessible to multiple clients.

[0031] As can be seen, the MAX-CAS instruction is very efficient in terms of latency and communication overhead: An alternative implementation would consist of first fetching the current value of the memory location over the network to NIC1, having NIC1 compare the current value with the comparison value, and, if necessary, sending the exchange value over the network for storage in the memory location. Hash Table Get / Set

[0032] In some embodiments, NIC1 and NIC2 support one or more RDMO commands that access a hash table in memory 40. Typically, NIC2 (the target NIC) is connected to a server that hosts the hash table in memory 40, and NIC1 (the initiator NIC) is connected to a client that accesses the hash table.

[0033] In the disclosed embodiment, the hash table is associated with a hash function that generates a hash value based on a key. Each hash value points to a location in the hash table. Each location in the hash table (pointed to by a corresponding hash value) comprises a linked list of zero or more {key, value} pairs corresponding to the hash value. If no value corresponding to a particular hash value is currently stored in the hash table, the linked list for that location in the hash table is empty.

[0034] A hash table fetch command instructs the target NIC to retrieve a value from the hash table, from a location in the hash table that matches a specified key. A hash table set command instructs the target NIC to write a new value to the hash table, at a location in the hash table that matches a specified key. In both cases, the command specifies the key. The target NIC computes a hash value by applying the hash function to the key and then accesses the location pointed to by the hash value to read or write the value.

[0035] Fig. Figure 3 is a flowchart schematically illustrating a method for executing a Hash Table Get RDMO command according to one embodiment of the present invention. The method begins with NIC1 (the initiator NIC) sending a Hash Table Get command over network 32 to NIC2 (the target NIC) at a command send stage 82. NIC2 receives the command over network 32 at a command receive stage 84.

[0036] At hash calculation level 86, NIC2 calculates a hash value by applying a hash function to the key specified in the command. The hash value points to a location in the hash table, which comprises a linked list.

[0037] At an element retrieval stage 88, NIC2 reads the next element (pair {key, value}) from the linked list stored at the location in the hash table pointed to by the hash value. (On the first iteration, NIC2 reads the head of the list, which may or may not be empty.)

[0038] At key verification level 90, NIC2 checks whether the key of the currently read element (pair {key, value}) matches the key specified in the command. If this is the case, NIC2 returns the value of the matching element to NIC1 over network 32 at a value return level, and the procedure is terminated.

[0039] If the key of the currently read element does not match the key specified in the command, NIC2 proceeds to check whether the linked list has been exhausted at a list check level 94. If this is the case, NIC2 returns an error message to NIC1 at an error level 96 over network 32, indicating that no value was found, and the procedure terminates. If the linked list has not yet been exhausted, the procedure loops back to level 88 above, and NIC2 proceeds to the next element in the linked list.

[0040] As with the MAX-CAS instruction, NIC2 typically executes stages 92, 94, and 102 atomically, meaning it does not allow any operations on the hash table between them. Operation atomicity is important, for example, when the hash table is accessible to multiple clients. Furthermore, it may be necessary to protect the hash table from other changes during the execution of the hash table fetch instruction. This type of locking can be accomplished in any suitable way.

[0041] The process in Fig. Figure 3 is an example flow chosen for clarity only. In alternative embodiments, any other suitable flow may be used. For example, a hash table set instruction may be executed in a similar manner.

[0042] The above procedures enable access to a remote hash table with low latency and minimal communication overhead: An alternative implementation would be to calculate the location in the table in NIC1 and then instruct NIC2 to access the linked list at the specified location (read or write). If the first access attempt fails, NIC1 would instruct NIC2 to retry and retrieve the next element in the linked list, and so on. This process would continue until it succeeds or until the linked list is exhausted. As can be seen, such a naive solution requires multiple round-trip transactions across the 32-bit network. Using RDMO for this use case thus reduces the sensitivity of hash table access to the number of collisions for the corresponding key. Insert into table / buffer (Table / Buffer Append)

[0043] Another type of RDMO instruction that can be supported by NIC1 and NIC2 is an instruction that inserts a new value at the end of a buffer stored in memory 40. A typical use case is inserting a value at the end of a table. Therefore, the terms "table" and "buffer" are used synonymously in this description. In addition to the table itself, memory 40 also stores a "write pointer"—a pointer that points to the memory location where the new value is to be inserted.

[0044] Fig. Figure 4 is a flowchart schematically illustrating a method for executing a Table Append RDMO command in accordance with one embodiment of the present invention. The method begins with NIC1 (the initiator NIC) sending a table append command over network 32 to NIC2 (the target NIC) at a command send stage 110. NIC2 receives the command over network 32 at a command receive stage 114.

[0045] At a pointer fetch stage 118, NIC2 fetches the table's write pointer from memory 40. At an insert stage 122, NIC2 appends the value specified in the instruction by writing the value to the location specified by the write pointer. At a pointer increment stage 126, NIC2 increments the write pointer. NIC2 typically performs stages 118, 122, and 126 atomically.

[0046] An alternative way to insert a value into a remote table would be to perform an atomic RDMA fetch-and-add operation on the write pointer across the network in NIC2's memory 40, returning the original value to NIC1, and then having NIC1 instruct NIC2 to write the new value to the location specified by the write pointer. The disclosed RDMO instruction reduces the additional round-trip transactions on the network and the associated latency. Fault-tolerant remote logging

[0047] Yet another use case that can benefit from the use of RDMO instructions is the logging of software transactions. Logging or journaling refers to any scheme that records actions performed by a software process, for example, to recover from a failure. In some embodiments, the logging functionality is offloaded to a network device (e.g., NIC), providing improved fault tolerance, among other benefits. Furthermore, the logging network device can log transactions running on remote hosts. The transactions are forwarded for logging using RDMO.

[0048] Fig. Figure 5 is a block diagram schematically illustrating a computer system 128 employing remote logging using RDMO according to one embodiment of the present invention. In system 128, host 28A (HOST1) executes a software process 130A labeled PROCESS1, and host 28B (HOST2) executes a software process 130B labeled PROCESS2.

[0049] NIC 24B (NIC2) includes a logger 134 that logs software transactions in memory 40. Logger 134 can log software transactions from PROCESS1 and / or transactions from PROCESS2. If a process (PROCESS1 or PROCESS2) fails (e.g., because the host crashed or for some other reason), logger 134 can recover the failed process from the log stored in memory 40. In one disclosed embodiment, NIC1 and NIC2 support an RDMO command that transfers one or more PROCESS1 transactions from NIC1 and NIC2 for logging by logger 134.

[0050] Fig. 6 is a flowchart schematically illustrating a method for remote logging using RDMO according to an embodiment of the present invention. The method begins with NIC1 sending a LOG RDMO command to NIC2 at a command send stage 138. The LOG command specifies (e.g., includes data and / or metadata from) a transaction of PROCESS1 and instructs NIC2 to log the transaction. At a command receive stage 142, NIC2 receives the LOG command over network 32. At a logging stage 146, the logger 134 in NIC2 logs the transaction to memory 40.

[0051] The Fig. 1 and Fig.The configurations of systems 20 and 128 illustrated in Figure 5, including the internal configurations of the network devices (e.g., NIC) and hosts within these systems, are example configurations chosen for conceptual clarity only. Any other suitable configurations may be used in alternative embodiments. Elements not necessary for understanding the principles of the present invention have been omitted from the figures for clarity.

[0052] Like the other RDMO instructions described herein, the LOG instruction is typically embedded in the transport protocol used between NIC1 and NIC2 (e.g., RDMA). NIC2 typically executes the instruction atomically in memory 40.

[0053] The various elements of systems 20 and 128, including the various disclosed network devices (e.g., NICs) and hosts, may be implemented in hardware, e.g., in one or more application-specific integrated circuits (ASICs) or FPGAs, in software, or using a combination of hardware and software elements. In some embodiments, certain elements of the disclosed network devices and / or hosts may be implemented, in whole or in part, with one or more general-purpose processors programmed in software to perform the functions described herein. The software may be downloaded to any of the processors in electronic form, e.g., over a network, or may alternatively or additionally be provided and / or stored on non-transitory tangible media, such as magnetic, optical, or electronic storage.

[0054] It should be appreciated that the embodiments described above are given as examples, and that the present invention is not limited to what has been particularly shown and described herein. Rather, the scope of the present invention includes combinations and sub-combinations of the various features described above, as well as variations and modifications thereof that would occur to those skilled in the art upon reading the foregoing description and that are not disclosed in the prior art. The documents incorporated by reference into this patent application are to be considered an integral part of this application, except that to the extent that terms in these incorporated documents are defined in a manner that conflicts with definitions made explicit or implicitly in this description, only the definitions in this description should be considered.

[0055] It goes without saying that the aspects and embodiments described above are only exemplary and that changes in detail may be made within the scope of the claims.

[0056] Each device, method, and feature disclosed in the description and (if applicable) in the claims and drawings may be presented independently or in any suitable combination.

[0057] The reference signs used in the claims are for illustrative purposes only and do not limit the scope of the claims. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] Maximum Compare-and-Swap Remote Direct Memory Operation (RDMO)", Attorney Docket Number 23-TV-1086US01

[0001] Hash Table Remote Direct Memory Operations (RDMO)", Attorney Docket Number 23-TV-1086US02

[0001] Remote Logging Remote Direct Memory Operations (RDMO)", Attorney Docket Number 23-TV-1086US04

[0001] Brock et al. in “RDMA vs. RPC for Implementing Distributed Data Structures”, Proceedings of the 2019 IEEE / ACM 9th Workshop on Irregular Applications: Architectures and Algorithms (IA3), November 2019

[0019]

Claims

[1] System comprising: a first network device for sending a command over a network specifying a value and instructing that the value be inserted into a set of values ​​in a memory; and a second network device for receiving the command over the network and for executing the command by inserting the value into the set of values. [2] The system of claim 1, wherein in response to the command, the second network device is arranged to read a pointer pointing to a memory location into which the value is to be inserted and to write the value into the memory location. [3] The system of claim 1 or 2, wherein the command is embedded in a transport protocol used by the first and second network devices. [4] The system of claim 3, wherein the transport protocol is a remote direct memory access (RDMA) protocol. [5] A system according to any one of the preceding claims, wherein the second network device is arranged to execute the instruction atomically. [6] Network device comprising: a network interface for connecting to a network; and processing circuitry for sending a command over the network specifying a value and instructing that the value be inserted into a set of values ​​in a memory. [7] The network device of claim 6, wherein the command is embedded in a transport protocol used by the network device. [8] The network device of claim 7, wherein the transport protocol is a remote direct memory access (RDMA) protocol. [9] Network device comprising: a network interface for connecting to a network; and processing circuitry for receiving a command over the network specifying a value and instructing that the value be inserted into a set of values ​​in a memory, and for executing the command by inserting the value into the set of values. [10] The network device of claim 9, wherein, in response to the command, the processing circuitry is arranged to read a pointer pointing to a memory location into which the value is to be inserted and to write the value into the memory location. [11] The network device of claim 9 or 10, wherein the command is embedded in a transport protocol used by the network device. [12] The network device of claim 11, wherein the transport protocol is a remote direct memory access (RDMA) protocol. [13] A network device according to any one of claims 9 to 12, wherein the processing circuitry is arranged to execute the instruction atomically. [14] Method comprising: Sending a command specifying a value and instructing that the value be inserted into a set of values ​​in a memory from a first network device over a network; and in a second network device, receiving the command over the network and executing the command by inserting the value into the set of values.

Images