Packet processing using programmable network interfaces

Abstract

A programmable network interface for a server includes: at least one memory storing connection parameters for multiple previously active network-structure-based non-volatile memory Express (NVMeoF) connections to different NVMe nodes. An NVMeoF connection request is received from an NVMe node, and it is determined whether the NVMe node is associated with the connection parameters stored in the at least one memory. In response to determining that the NVMe node is associated with the connection parameters stored in the at least one memory, a new NVMeoF connection is established for communicating with the NVMe node using the stored connection parameters. In one aspect, the server's address space is partitioned, and an NVMe request queue is allocated to each partition of the address space. At least one address in a received NVMeoF message is identified, and an NVMe request queue for executing the NVMe request included in the NVMeoF message is determined.

Description

[0001] Cross-reference to related applications

[0002] This patent application claims priority to U.S. Provisional Patent Application No. 63 / 124,597 (Attorney General's No.: WDA-5230P-US), filed December 11, 2020, entitled "Packet Processing by Programmable Network Interface," which is incorporated herein by reference in its entirety. This patent application also claims priority to U.S. Patent Application No. 17 / 172,513 (Attorney General's No.: WDA-5230-US), filed February 10, 2021, entitled "Packet Processing by Programmable Network Interface," which is incorporated herein by reference in its entirety. Background Technology

[0003] The current trends of cloud computing, big data, and I / O-intensive applications are driving increased demand for high-performance distributed shared memory systems in data centers, requiring low latency, high throughput, and high bandwidth. Emerging high-performance non-volatile memory, Express (NVMe), is a key driver of this demand. ® Storage devices, such as solid-state drives (SSDs), are becoming increasingly common due to their improved performance in terms of input / output operations per second (IOPS). While NVMe was originally designed as an interface specification to be used with the Peripheral Component Interconnect Express (PCIe) interface within computer systems, the latest NVMe extensions, such as network-based NVMe (NVMeoF), can support remote access to NVMe devices, such as Remote Direct Memory Access (RDMA) (e.g., RDMA over Aggregated Ethernet (RoCE) and iWarp) and Fibre Channel (FC) transport architectures.

[0004] While both RoCe and iWARP support high performance, they require RDMA-enabled devices and expensive Data Center Bridging (DCB)-enabled switches, which may be difficult to obtain and limit scalability. On the other hand, NVMeoF using Transmission Control Protocol (TCP) allows NVMe to run on existing Ethernet networks in data centers using the most prevalent transport protocols. However, the efficiency of such NVMeoF systems using TCP depends on the performance of the TCP stack. In this respect, TCP packet processing can be processor-intensive, leading to significant performance penalties, resulting in higher latency and lower throughput. While network bandwidth continues to increase, the inefficiency of standard network interfaces used for network processing (e.g., Network Interface Cards (NICs)) limits data center memory and computing performance.

[0005] In practice, kernel-space TCP stack operations consume the majority of central processing unit (CPU) cycles for stateful TCP operations and connection placement. This can lead to significant processing overhead and performance degradation, especially for relatively short connections and for latency-sensitive streams (e.g., "bursting mouse streams") that are common in data centers and consist of only a few packets.

[0006] In some cases, kernel bypass can be used to offload packet processing from the server's kernel space to applications executing in the server's user space. However, kernel bypass typically still cannot keep up with modern Ethernet speeds, such as 100 Gbps or 200 Gbps, or support the per-stream resource allocation requirements of today's data centers. Despite using kernel bypass, the improvement in TCP performance is limited, primarily due to the overhead of complex protocol operations for maintaining the protocol specification. Attempts have also been made to offload some basic operations, such as checksum processing and packet segmentation, from the server's processor; however, the kernel's TCP stack processing often remains a performance bottleneck. Furthermore, these methods do not provide flexible offloading of performance-dependent packet processing operations. In particular, these methods do not support rapid adaptation to the evolution of different protocols or the ever-changing needs of applications executing on today's data center nodes. Attached Figure Description

[0007] The features and advantages of the embodiments of this disclosure will become more apparent from the detailed description below, taken in conjunction with the accompanying drawings. The drawings and associated description are provided to illustrate embodiments of this disclosure and not to limit the scope of the claims.

[0008] Figure 1 An exemplary network environment for implementing packet processing offloading is shown according to one or more embodiments.

[0009] Figure 2It is a block diagram of exemplary components included in a programmable network interface according to one or more embodiments.

[0010] Figure 3 It is an exemplary programmable pipeline of a programmable network interface according to one or more implementations.

[0011] Figure 4 It is an exemplary network-based non-volatile memory Express (NVMeoF) / transmission control protocol (TCP) connection established according to one or more implementation schemes.

[0012] Figure 5 It is an exemplary NVMeoF write transaction based on one or more implementation schemes.

[0013] Figure 6 It is a flowchart of the NVMeoF / TCP connection establishment process according to one or more implementation schemes.

[0014] Figure 7 It is a flowchart of the NVMe request management process according to one or more implementation schemes.

[0015] Figure 8 It is a flowchart of the NVMeoF message retransmission process according to one or more implementation schemes. Detailed Implementation

[0016] Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that various embodiments disclosed may be practiced without some of these specific details. In other instances, well-known structures and techniques have not been shown in detail to avoid unnecessarily obscuring the various embodiments.

[0017] Exemplary network environment

[0018] Figure 1 An exemplary network environment 100 for implementing packet processing offloading is shown according to one or more embodiments. Figure 1 As shown, nodes 104A, 104B, and 104C communicate with each other via network 102. Node 104 may include a server, which may serve as a processing node, for example, including one or more processors, such as a Reduced Instruction Set Computer (RISC)-V core, and / or serve as a memory node, including one or more solid-state drives (SSDs) or hard disk drives (HDDs). In some implementations, Figure 1The network environment 100 can be used as at least part of a data center and / or for distributed processing, such as distributed machine learning or big data analytics. As those skilled in the art will understand, node 104 is shown for illustrative purposes, and the network environment 100 may include more than Figure 1 The diagram shows more nodes. Furthermore, those skilled in the art will understand that network environment 100 may include more than [the number of nodes shown]. Figure 1 The components shown include more components such as clustered switches, top-of-rack (ToR) switches, and network controllers.

[0019] Nodes 104 communicate with each other via network 102. Network 102 may include, for example, a Storage Area Network (SAN), a Local Area Network (LAN), and / or a Wide Area Network (WAN), such as the Internet. In this respect, nodes 104A, 104B, and 104C may not be located in the same geographical location. Nodes 104 may communicate using one or more standards such as Ethernet. In this respect, connections 114A, 114B, and 114C connect programmable network interfaces 112A, 112B, and 112C to network 102, respectively. Programmable network interface 112 may include one or more Ethernet ports for connecting 114.

[0020] Each node in node 104 includes one or more processors 106, memory 108, and a programmable network interface 112. These components of node 104 can communicate with each other via bus 110, which may include, for example, a Peripheral Component Interconnect Express (PCIe) bus. Node 104 may be referred to herein as a network-based non-volatile memory Express (NVMeoF) node, wherein node 104 is configured to communicate with other nodes 104 using NVMe messages (e.g., NVMe commands and responses), which may be encapsulated in Ethernet packets using Transmission Control Protocol (TCP) capsules.

[0021] Processor 106 can execute instructions, such as instructions from an application executing in the user space of node 104, or instructions that are part of an operating system (OS) used by node 104. Processor 106 may include circuitry such as a central processing unit (CPU), graphics processing unit (GPU), microcontroller, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), one or more RISC-V cores, hard-wired logic, analog circuitry, and / or combinations thereof. In some implementations, processor 106 may include a system-on-a-chip (SoC) that can be combined with memory 108.

[0022] Memory 108 may include, for example, volatile random access memory (RAM) such as static RAM (SRAM), dynamic RAM (DRAM), non-volatile RAM, or other solid-state memory used by one or more processors 106. Data stored in memory 108 may include data read from another node 104, data to be stored in another node 104, instructions loaded from an application for execution by processor 106, and / or data for executing such an application.

[0023] In some implementations, memory 108 may include, for example, one or more spinning disks or non-volatile solid-state memories, such as flash memory. In such implementations, memory 108 may include one or more storage devices, such as one or more SSDs and / or HDDs. Although the description herein generally refers to solid-state memory, it should be understood that solid-state memory may include one or more of a variety of memory devices, such as flash memory integrated circuits, NAND memory (e.g., single-cell (SLC) memory, multi-cell (MLC) memory (i.e., two or more layers) or any combination thereof), NOR memory, EEPROM, other discrete non-volatile memory (NVM) chips, or any combination thereof. In other implementations, for example, memory 108 may include storage-class memory (SCM), such as chalcogenide RAM (C-RAM), phase-change memory (PCM), programmable metallized cell RAM (PMC-RAM or PMCm), bidirectional universal memory (OUM), resistive RAM (RRAM), ferroelectric RAM (FeRAM), magnetoresistive RAM (MRAM), 3D-XPoint memory, and / or other types of solid-state memory.

[0024] The programmable network interface 112 may include devices such as a smart network interface card (NIC), a network interface controller, or a network adapter that can be programmed using a programming language such as P4. For example, the programmable network interface 112 may include a smart NIC, such as a Netronome Agilio NIC, a Mellanox Bluefield NIC, or a Broadcom Stingray NIC. Figure 1 As shown, each programmable network interface 112 stores a connection management module 10, a transmission module 12, a flow control module 14, a request queue 16, and connection parameters 18.

[0025] As discussed in more detail below, the connection management module 10, transmission module 12, and flow control module 14 can enable NVMe and TCP processing offloading from the processor 106 to the programmable network interface 112. This offloading can allow node 104 to achieve higher performance in terms of reduced memory access latency and increased throughput. Request queue 16 can allow queuing of NVMe requests or commands executed in different partitions of the accessible address space of node 104. The address space can represent the amount of storage devices or memory at node 104 that can be accessed by other nodes 104, such as the amount in memory 108.

[0026] Additionally, the programmable network interface 112 stores connection parameters 18, including connection parameters for previous active connections with different nodes 104. Such connection parameters may include, for example, indications of connection status, negotiation parameters between nodes 104 during previous active connections, packet sequence numbers, and / or retransmission timeout values ​​for retransmitting messages if no acknowledgment is received from other nodes 104. Negotiation parameters may include, for example, the protocol version used for previous active connections, data offset alignment with the start of Protocol Data Units (PDUs) used in NVMe, a bitmask for the header or a data digest for ensuring data integrity, maximum data length, permission levels (e.g., read-only, read / write permissions), and / or the maximum number of credits allowed for flow control. As discussed in more detail below, storing connection parameters 18 at the programmable network interface 112 can reduce the time required to establish new connections between nodes by reusing stored connection parameters from previous active connections between the same nodes 104. In some implementations, connection parameters for less new and / or less frequent connections may be removed to free up space for connection parameters 18 for newer and / or more frequent connections.

[0027] Those skilled in the art will understand from this disclosure that, Figure 1 Compared to the example shown, other implementations could include different numbers or arrangements of nodes 104. In this respect, Figure 1 The network environment 100 shown is for illustrative purposes, and those skilled in the art will understand that the network environment 100 may include more than Figure 1 The example shows more nodes 104 and additional components such as routers and switches.

[0028] Figure 2 This is a block diagram of exemplary components included in a programmable network interface 112A according to one or more embodiments. Figure 2 As shown, the programmable network interface 112A includes port 116. 1A and 116 2A Circuit 118A, memory 120A and interface 122A.

[0029] Port 116 1A and 116 2A Allows the programmable network interface 112A to be connected via 114 1A and 114 2A It communicates with other nodes 104. In other implementations, the programmable network interface 112A may include a different number of ports 116, such as a single port 116.

[0030] Circuit 118A can execute instructions, such as those from connection management module 10A, transmission module 12A, and flow control module 14A. Connection management module 10A, transmission module 12A, and flow control module 14A may include firmware and / or field-programmable instructions for programmable network interface 112A. As described above, programmable network interface 112A can be field-programmed using programming languages ​​such as P4. This allows programmable network interface 112A to adapt to new versions of protocols used in network environment 100, and / or change requirements for applications executed by node 104, such as minimum quality of service (QoS). Circuit 118A may include circuitry such as a microcontroller, DSP, ASIC, FPGA, multi-core processor, hardwired logic, analog circuitry, and / or combinations thereof. In some implementations, circuit 118A may include a System-on-a-Chip (SoC), which may be combined with memory 120A or other memory such as DRAM.

[0031] Memory 120A may include, for example, content-addressable memory (CAM) or a ternary CAM (TCAM) table, or other types of match action tables, which typically provide faster message processing than conventional network interfaces (e.g., NICs), such as message processing at line rate or near line rate. In some implementations, the CAM or TCAM table may be stored in registers, such as an SRAM register array. In other implementations, memory 120A may include solid-state memory, such as flash memory or SCM. Figure 2 As shown, the memory 120A stores the connection management module 10A, the transmission module 12A, the flow control module 14A, the request queue 16A, and the connection parameters 18A.

[0032] The connection management module 10A is responsible for NVMeoF / TCP connection establishment, connection maintenance, and connection teardown. (See below for reference.) Figure 4 In more detail, the programmable network interface 112A can communicate with the programmable network interface 112 of another node 104 (e.g., Figure 1The programmable network interface 112B in the node exchanges connection management messages to establish an NVMeoF / TCP connection, and then notifies one or more processors 106A of the node 104A that the connection has been established. After receiving a disconnection request from the node or from one or more processors 106A executing the application, the connection management module 10A performs NVMeoF / TCP connection teardown.

[0033] Furthermore, the connection management module 10A of the programmable network interface 112A is also responsible for discovering and identifying NVMe subsystems or NVMe nodes 104 accessible to node 104. NVMe discovery may include an identification path for communicating with each discovered NVMe subsystem and NVMe node 104. In some implementations, the programmable network interface 112A using the connection management module 10A may broadcast a discovery message during the initialization of node 104A. Each NVMe node 104 receiving the discovery message may respond using its available NVMe subsystem information (e.g., NVMe subsystem name and address). The connection management module 10A then creates a data structure in the memory of the programmable network interface 112A (e.g., memory 120A) that includes information for node 104A to connect to different NVMe subsystems based on information collected from other NVMe nodes 104.

[0034] The transport module 12A is responsible for the transmission of data and commands between nodes 104 during NVMe read and write operations. This may include identifying NVMe commands encapsulated in incoming NVMeoF / TCP packets, removing Ethernet, TCP, and / or NVMe headers, and sending only the operation code and required parameters to the request queue in request queue 18A based on at least one address identified by the transport module 12A for the command. In other words, the transport module 12A can translate NVMeoF / TCP packets into actual read and write commands executed by one or more processors 106A of node 104A.

[0035] Furthermore, the transmission module 12A can maintain a data structure indicating the status of each running transaction (i.e., each pending NVMe command sent to the target node or received from the initiator node) in the memory 120A. In some implementations, and as referenced below... Figure 5 The example of fragmented NVMeoF write transactions is discussed in more detail. The transport module 12A can segment or fragment large NVMeoF / TCP data transmissions, where the NVMe PDU is larger than the network maximum transmission unit (MTU), such as 1,500 bytes.

[0036] The flow control module 14A is responsible for load balancing, congestion control, controlling NVMe retransmission timeout values, and NVMe capsule retransmission. In some implementations, the flow control module 14A controls the address space of the memory of node 104A that can be accessed by other nodes 104 (e.g., Figure 1 The memory 108A is partitioned (some or all of its address space), and an NVMe request queue is allocated to each partition of the address space to form request queue 18A. The NVMe request or command is then routed to the appropriate queue based on at least one address included in the NVMeoF message. The flow control module 14A can also balance the queues by setting different weights or numbers of requests to execute each queue in a given round (e.g., in a round-robin algorithm). The weights or numbers of requests may also depend on application execution requirements (e.g., minimum QoS) and / or, for example, the priority indicated for the NVMe request by using an 802.1Qbb priority tag included in the packet.

[0037] Furthermore, the flow control module 14A can use a credit-based system to help ensure sufficient buffer space for receiving NVMe commands and data capsules. For example, based on the available capacity of the NVMe request queue used by the initiator, the programmable network interface 112A can use the flow control module 14A to send an indication of multiple credits to the initiator node 104 for sending a message to node 104A. Then, whenever a node sends an NVMeoF message to node 104A, the initiator node 104 decreases its credit count, for example, by one. Credits can be increased in response to new indications of credits returned by node 104A to other nodes 104.

[0038] The flow control module 14A can also use the Explicit Congestion Notification (ECN) bit in one or more packets to identify network congestion. In response to the ECN bit being set, the programmable network interface 112A can notify the sender in the next acknowledgment packet to adjust its transmission rate.

[0039] To support reliable NVMe capsule delivery, the flow control module 14A can use selective acknowledgment and timeout retransmission methods. Upon receiving a packet, the flow control module 14A can compare the sequence number of the received packet with the expected sequence number of the data stream or active connection. If the sequence number of the received packet matches the expected sequence number, the programmable network interface 112A further processes the packet and returns an acknowledgment (i.e., ACK) to the sender. Otherwise, the programmable network interface 112A sends a negative acknowledgment (i.e., NACK) or other error message, including the sequence number of the last packet received by node 104A.

[0040] The flow control module 14A can also retransmit packets if a negative acknowledgment is received by the programmable network interface 112A or if no acknowledgment is received within a time period, such as a retransmission timeout period (RTO), represented by a timeout value. In some implementations, the flow control module 14A can set a timeout register for receiving corresponding acknowledgments in response to, for example, sending a message through the egress pipeline of the programmable network interface 112A. The timeout value can be based on, for example, the average round-trip packet transmission duration. In some implementations, the timeout value can be adjusted in the field based on, for example, an exponentially moving average of sampled round-trip times.

[0041] Interface 122A provides an interface to the internal bus 110A of node 104A, such as Figure 1 As shown. In some implementations, interface 122A may provide a PCIe interface or PCIe switching for communication with other components of node 104A, such as with... Figure 1 One or more processors 106A and memory 108A communicate within the system.

[0042] As will be understood by those skilled in the art from this disclosure, the arrangement or number of components of the programmable network interface 112A can be related to... Figure 2 Different. For example, other implementations of the programmable network interface 112A may include different numbers of memories 120A, interfaces 122A, or different numbers of ports 116.

[0043] Figure 3 This is an exemplary programmable pipeline 20A for a programmable network interface 112A according to one or more embodiments. Using a programmable pipeline in a node's programmable network interface allows for the efficient offloading of tasks typically performed by virtualization or by drivers (e.g., NVMe drives) from the node's processors. This hardware-based offloading of NVMeoF / TCP processing from one or more processors of the node to the programmable network interface typically increases throughput and reduces latency to better utilize NVMeoF and modern Ethernet speeds. In some implementations, each port of the programmable network interface, such as port 116... 1A and 116 2A Each port in the pipeline can have its own programmable pipeline 20A for processing packets received by the port at line rate or near line rate.

[0044] like Figure 3 As shown in the example, the programmable pipeline 20A includes a parser 22A, one or more matching action tables 24A, and an inverse parser 26A. Figure 3In the example, the ingress packet is received by parser 22A, which extracts the header from the packet to perform a matching action operation or phase using one or more matching action tables 24A. Each matching action table 24A can be customized to perform a matching operation on one or more extracted values ​​from the packet according to predefined rules, and then perform actions such as adding, removing, or modifying header fields, routing NVMe requests to the appropriate queue 18 based on the extracted address and / or priority indicator, or sending a connection management response, such as a message for adjusting the transmission rate due to the extracted ECN bit, a message for establishing a connection, or a message for terminating an active connection. The de-parser 26A can assemble packets in response to the following without involving other components of node 104A outside the programmable network interface 112A (e.g., Figure 1 One or more processors 106A in the process: received packets, such as acknowledgments (ACK messages), error messages (e.g., NACK messages), or other notifications, such as notifications for adjusting the transmission rate or the number of available credits for flow control.

[0045] Exemplary process

[0046] Figure 4 This is an exemplary NVMeoF / TCP connection established between nodes 104A and 104B according to one or more implementation schemes. For example... Figure 4 As shown, connection establishment can be performed by the programmable network interfaces 112A and 112B of nodes 104A and 104B, which respectively execute connection management modules 10A and 10B. In some implementations, almost all communication exchanged between programmable network interfaces 112A and 112B can be performed by the corresponding programmable pipeline (e.g., programmable pipeline 20A discussed above) with minimal involvement from other components of nodes 104A and 104B, which improves the speed at which NVMeoF / TCP connections can be established.

[0047] like Figure 4 As shown, connection establishment can begin with a TCP three-way handshake, where programmable network interface 112A sends a TCP synchronization packet (i.e., TCP Syn) that may include a sequence number to programmable network interface 112B. In response, programmable network interface 112B returns a TCP synchronization packet, acknowledging receipt of the TCP synchronization packet from programmable network interface 112A (i.e., TCP Syn ACK), for example by incrementing the received sequence number and providing its own sequence number to programmable network interface 112A. Connection establishment acknowledgment (i.e., ACK) can, for example, increment the sequence number provided by programmable network interface 112B.

[0048] Then, programmable network interface 112A performs NVMe / TCP connection establishment by sending an NVMe / TCP connection initialization request to programmable network interface 112B. This may include a PDU sent from programmable network interface 112A to provide NVMe / TCP connection parameters, such as features supported by node 104A and / or parameters to be negotiated between programmable network interfaces 112A and 112B.

[0049] Such negotiation parameters may include, for example, the protocol version to be used, the data offset to be used in the PDU, a bitmask for the header or a data digest to ensure the integrity of the PDU, and the maximum data length for each PDU. In response, the programmable network interface 112B returns an NVMe / TCP connection initialization response, which may confirm the features and / or parameters proposed by the programmable network interface 112A and may include additional information, such as the maximum credit count that the programmable network interface 112A can use to send messages. As described above, the flow control module of the programmable network interface 112B can set the maximum credit count based on the buffer or queue size of the programmable network interface 112B.

[0050] After establishing an NVMe / TCP connection, the programmable network interface 112A establishes an NVMeoF connection by sending an NVMeoF connection request. The NVMeoF connection request may include information such as that used for NVMeoF binding, such as limitations on certain capabilities (e.g., hash list (SGL) descriptors for large data transfers, data placement limitations, data transfer capabilities, and authentication capabilities). The NVMeoF connection request may also include information such as that about the queues used by the programmable network controller, such as the size and number of queues. The programmable network interface 112A may notify the processor 106A of the node 104A that a connection has been established, enabling commands to be transmitted to node 104B after the programmable network interface 112A has established the NVMeoF / TCP connection. The processor 106A may request the connection by, for example, executing an application that requests data stored in the NVMe namespace of node 104B or requests data to be written to the NVMe namespace of node 104B.

[0051] As described above, using a programmable network interface to establish an NVMeoF / TCP connection can significantly reduce the amount of time required to establish an NVMeoF / TCP connection between nodes 104A and 104B, while also offloading such processing from processor 106 to further improve the performance of node 104. Similarly, connection management module 10 can also be used for NVMeoF / TCP connection teardown. In this case, programmable network interface 112 can receive a disconnect request from other NVMe nodes or from an application executing at node 104, and send a completion message to other NVMe nodes without involving other components of node 104. A notification indicating that the connection is no longer active can then be provided to processor 106 of node 104.

[0052] Additionally, programmable network interfaces 112A and 112B can retain connection parameters after connection termination. These parameters can be stored as part of connection parameters 18 to facilitate faster connection establishment for future NVMeoF / TCP connections between programmable network interfaces 112A and 112B. In this regard, if the connection parameters stored in connection parameters 18 can be identified for other NVMe nodes, including other programmable network interfaces, then skipping [the necessary steps] is possible. Figure 4 Some or all of the NVMe / TCP connection establishment and NVMeoF connection establishment processes.

[0053] refer to Figure 4 In this sequence, the programmable network interface 112B can use connection parameter 18A to preemptively send an NVMe / TCP connection initialization response to confirm the connection parameters used for the last active connection with the programmable network interface 112A, and can also preemptively send an NVMeoF connection response to confirm other connection parameters used for the last active connection with the programmable network interface 112A. See below for reference. Figure 6 The connection establishment process is discussed in more detail. Reusing stored connection parameters from previous connections to the same node can often reduce the time and processing required to establish an NVMeoF / TCP connection.

[0054] Figure 5 This is an exemplary NVMeoF write transaction fragmented according to one or more implementation schemes. For example... Figure 5 As shown, the programmable network interface 112A can fragment the data received from the processor (i.e., processor 106A) of node 104A to fit the network MTU. Figure 5 The fragmented write sequence can be primarily executed by the programmable network interfaces 112A and 112B of nodes 104A and 104B, which respectively execute transmission modules 12A and 12B. In some implementations, in Figure 5Almost all communication exchanged between the programmable network interfaces 112A and 112B can be performed by the corresponding programmable pipeline, such as the programmable pipeline 20A discussed above, with almost no involvement of other components of nodes 104A and 104B. This improves the speed at which large data transfers (i.e., larger than MTU) can be performed.

[0055] The programmable network interface 112A begins by sending an NVMe write command capsule in the PDU after an NVMeoF / TCP connection has been established, as discussed above. Figure 4 As shown in the sequence, or by using connection parameter 18, as referenced below. Figure 6 The NVMeoF / TCP connection establishment process is discussed in more detail. From Figure 5 The NVMe write command capsule sent by the programmable network interface 112A in the capsule includes NVMe write commands.

[0056] In response to receiving an NVMe write command capsule, the programmable network interface 112B returns a ready-to-transmit PDU. As described above, the programmable controller, via the connection management module 10A, can provide the programmable network interface 112A with a maximum credit count of multiple messages or PDUs that can be sent before receiving an acknowledgment from the programmable network interface 112B. In other implementations, the ready-to-transmit PDU may include a credit count customized for a specific queue allocated to the portion of the address space where the write command will be executed, which may be indicated by one or more addresses included in the initial NVMe write command capsule.

[0057] Then, the programmable network interface 112A connects to the processor of its slave node (e.g., Figure 1 The processor 106A receives data, fragments it, and packages the data to be written into a series of data PDUs. In this regard, the transmission module 12A can provide a sequence number for each PDU to ensure data is received in the correct order. The transmission module 12A can include an indicator or marker in the last data PDU to indicate that transmission is complete.

[0058] After receiving the last data PDU, programmable network interface 112B returns a response capsule PDU to programmable network interface 112A indicating that all data PDUs have been received. In this respect, the transmission module 12B, executed by programmable network interface 112B, may include an indication or flag in the response capsule PDU indicating that all data PDUs were received in the correct order. In other implementations, the response capsule PDU may be omitted, such that programmable network interface 112A assumes that all data PDUs were successfully received in the correct order unless an error message or NACK is received from programmable network interface 112B.

[0059] In some implementations, the transmission module 12B may send an error message or NACK in response to a data PDU not being received in the correct order. In such cases, and as referenced below... Figure 8 The exemplary message retransmission process is discussed in more detail. The transmission module 12A of the programmable network interface 112A can retransmit out-of-order or lost data PDUs with minimal involvement from other components of the node 104A (e.g., the processor 106A of the node 104A).

[0060] During active connections or communication sessions between nodes, in some implementations, each programmable network interface 112 can track the current state of each transaction, where commands have been sent to or received by another NVMe node. In some implementations, such information can be stored in a different data structure than the connection parameters 18 or in the same data structure.

[0061] As described above, using programmable network interfaces, such as programmable network interfaces 112A and 112B, can significantly reduce the amount of time required to transfer data using NVMeoF / TCP between nodes 104A and 104B, while also offloading such processing from processor 106 to further improve the performance of node 104.

[0062] Figure 6 It is a flowchart of the NVMeoF / TCP connection establishment process according to one or more implementation schemes. Figure 6 The process can be performed by, for example, circuit 118 of programmable network interface 112 of connection management module 10.

[0063] In box 602, the programmable network interface is located from the NVMe node (e.g., from...). Figure 1 Another node (104) receives the NVMeoF connection request. In some implementations, the connection request may follow, for example, TCP connection establishment and / or NVMe / TCP connection establishment, such as... Figure 4 The exemplary connection sequence is shown.

[0064] In box 604, the programmable network interface determines whether the NVMe node is associated with the connection parameters stored in connection parameters 18. In some implementations, an NVMe namespace ID or other identifier may be used to determine whether an entry or connection parameter is associated with the NVMe node. As described above, the programmable network interface may locally store parameters from previous NVMeoF / TCP connections, such as indications of connection status, negotiation parameters between programmable network interfaces 112 during previous active connections, packet sequence numbers, and / or retransmission timeout values ​​for retransmitting messages if no acknowledgment is received from other nodes 104. Negotiation parameters may include, for example, the protocol version used for previous active connections, data offset alignment with the start of the PDU, a bitmask for the header, maximum data length, permission levels (e.g., read-only, read / write permissions), and / or the maximum number of credits allowed for flow control.

[0065] In response to determining in box 604 that the NVMe node is associated with the connection parameters stored in connection parameters 18, a new NVMeoF connection is established in box 606 to communicate with the NVMe node using the stored connection parameters. (See above reference) Figure 4 As described in the connection establishment sequence, most NVMe / TCP connection establishment and NVMeoF connection establishment can replace the connection parameters of the storage associated with the NVMe node. In some implementations, the programmable network interface 112 can send acknowledgments of the storage parameters; however, even in this case, other NVMe nodes may not need to send their NVMe / TCP connection initialization requests and / or NVMeoF connection requests, which can reduce the amount of processing required to establish an NVMeoF / TCP connection.

[0066] On the other hand, in response to determining in box 604 that the NVMe node is not associated with the connection parameters stored in connection parameters 18, in box 608, the programmable network interface creates one or more new entries in connection parameters 18 to store the connection parameters of other NVMe nodes. For example, after establishing an NVMeoF / TCP connection, as discussed above... Figure 4 After the connection establishment sequence is completed, connection parameters for other NVMe nodes can be stored. As mentioned above, in some implementations, the storage of connection parameters may be limited by available storage space, such that only the connection parameters of the most recent and / or most frequently used connections are stored or retained in connection parameters 18. In this regard, connection parameters 18 may also include the last activity time and / or usage indicator for the established connection, which can be used to determine whether to remove connection parameters that are not recently used and / or less frequently used.

[0067] As described above, storing connection parameters 18 at the programmable network interface 112 can shorten the time required to establish a new NVMeoF / TCP connection between nodes by reusing stored connection parameters from previously active connections between the same nodes 104.

[0068] Figure 7 It is a flowchart of the NVMe request management process according to one or more implementation schemes. Figure 7 The process can be performed by, for example, circuit 118 of programmable network interface 112 of transmission module 12.

[0069] In block 702, programmable network interface 112 partitions the address space of its node 104. As described above, node 104 may include, for example... Figure 1 The server has at least one memory or one or more storage devices (e.g., HDD and / or SSD) in its memory 108. The programmable network interface can partition the address space representing the memory or storage device, which can be accessed by other NVMe nodes as an NVMe namespace, for example by partitioning the address space into a predetermined number of address ranges of equal size.

[0070] In box 704, programmable network interface 112 assigns an NVMe request queue to each partition from box 702. The partitioning and assignment of NVMe request queues can be performed during node setup or initialization or during the offline phase, as shown by the dashed lines below box 704.

[0071] During operation, in box 706, the programmable network interface receives an NVMeoF message from another NVMe node. The NVMeoF message may include NVMeoF and TCP packet headers that are parsed by the programmable network interface to perform matching actions. For example, by recognizing the operation code within the NVMe capsule contained in the NVMeoF message, the programmable network interface can identify the NVMe request embedded in the NVMeoF message. The operation code may indicate a read command, for example, to read data from an address in the NVMe namespace of node 104 (e.g., a server), or a write command, to write data to the NVMe namespace of node 104.

[0072] In block 708, the programmable network interface identifies at least one address in the received NVMeoF message for accessing data in the NVMe namespace of node 104. After parsing the NVMeoF message, the programmable network interface can remove some or all of the Ethernet, TCP, and NVMe headers included in the NVMeoF message. The programmable network interface 112 can then provide read or write commands only to the processor 106 of node 104 or other components with other NVMe parameters, effectively translating the received NVMeoF message into NVMe read or write commands to be executed by node 104. This translation by the programmable network interface 112 reduces the processing overhead otherwise required by the processor of node 104, thereby freeing up processor resources and improving the overall performance of node 104.

[0073] In block 710, programmable network interface 112 determines an NVMe request queue from the NVMe request queue allocated in block 704 based on the address identified in block 708, for executing the NVMe request included in the NVMeoF message. Programmable network interface 112 can identify the NVMe request queue using a matching action table, for example, matching an identified address within an address range corresponding to a partition of the NVMe namespace. The queue can be buffered or temporarily stored in the on-chip memory of programmable network interface 112 (e.g., write and read commands), for example in... Figure 2 The NVMe requests are then placed in the request queue 16. The programmable network interface 112, via the flow control module 14, can then load balance the performance of NVMe messages in the NVMe request queue 18 by sending them in a specific order or sequence to the bus 110 of node 104, for example, using a round-robin algorithm. Such load balancing via the programmable network interface 112 can often help prevent large data flows from blocking smaller data flows, which may be time-sensitive or otherwise require minimum QoS for data access.

[0074] In some implementations, the matching action table of the programmable network interface 112 can be programmed to identify and consider priority values ​​(e.g., 802.1Qbb values) included in the NVMeoF message when determining the NVMe request queue for an NVMe request. Such priority values ​​can be used by the NVMe node sending the NVMeoF message to associate certain NVMeoF messages with different applications that generate underlying NVMe requests that may have different QoS requirements (e.g., maximum latency for accessing data). In such implementations, the programmable network interface 112 can have a separate queue that can schedule NVMe messages more frequently or at a greater number per cycle than other lower-priority queues used for NVMe messages not originating from NVMeoF messages that include high-priority values.

[0075] Figure 8 It is a flowchart of the message retransmission process according to one or more implementation schemes. Figure 8 The process can be performed by, for example, circuit 118 of programmable network interface 112 of flow control module 14.

[0076] In block 802, programmable network interface 112 sends NVMeoF messages to the NVMe node. NVMeoF messages may originate from an application executed by processor 106 of node 104, which includes programmable network interface 112. The programmable network interface may add headers, such as TCP, Ethernet, and NVMeoF headers, to the NVMe messages received from processor 106 via bus 110 of node 104.

[0077] In box 804, programmable network interface 112 temporarily stores data, such as payload, from NVMeoF messages sent to another NVMe node. The data from the messages can be stored locally at programmable interface 112, for example, stored in [a specific location / system]. Figure 2 The payload or data of the NVMeoF messages sent by the programmable network interface 112 is stored in memory 120A. By storing this payload or data, the NVMeoF messages can typically be retransmitted more quickly when needed, such as after receiving an error message or NACK, or after the timeout value for receiving an acknowledgment has expired, as described below. The programmable network interface 112 can handle such errors and retransmissions without involving other components of node 104, such as processor 106, to further improve the overall performance of node 104.

[0078] In block 806, programmable network interface 112 determines whether a timeout value has been reached for receiving an acknowledgment of an NVMeoF message sent to another NVMe node. In some implementations, programmable network interface 112 may set a timeout register for receiving the corresponding acknowledgment in response to, for example, sending an NVMeoF message through an egress pipeline of the programmable network interface. The timeout value may be based on, for example, the average round-trip packet transmission duration. In some implementations, the timeout value may be field-adjusted based on, for example, an exponentially moving average of sampled round-trip times.

[0079] In response to a timeout value being reached in box 806 without receiving an acknowledgment, programmable network interface 112 retransmits the NVMeoF message to another NVMe node in box 808 using data or a payload temporarily stored in box 804. As described above, programmable network interface 112 can also retransmit the NVMeoF message in response to receiving an error message or NACK from another NVMeoF. The retransmission of the NVMeoF message can be handled entirely by programmable network interface 112 without involving other components of node 104.

[0080] In response to receiving an acknowledgment in box 806 before reaching a timeout value, programmable network interface 112 may send a next NVMeoF message in box 810. The next NVMeoF message may include an NVMe request from an application executing on node 104, or may include, for example, a "keep active" message to determine whether connections to other NVMe nodes remain active. In this respect, programmable network interface 112 may periodically send such keep active messages to maintain or test connections to other NVMe nodes without involving other components of node 104, such as the node's processor 106.

[0081] If no keep-alive message is acknowledged or if another NVMe node sends a disconnect request, the programmable network interface 112 performs connection teardown, for example, by releasing resources used to maintain the NVMeoF / TCP connection. In some implementations, if the connection to other NVMe nodes is reactivated, one or more values ​​used to maintain metadata about the connection (e.g., the last sequence number of the connection) can be stored as part of connection parameter 18 for future use. As described above, the programmable network interface can reduce the time and processing required to re-establish the NVMeoF / TCP connection by using connection parameter 18.

[0082] As mentioned above, offloading NVMeoF / TCP processing to a programmable network interface can often improve the performance of nodes such as servers, since packet processing, like TCP packet handling, can be processor-intensive. In this regard, many operations of such NVMeoF / TCP processing can be handled at line rate or near line rate through the matching action phase of such a programmable network interface. Furthermore, the programmability of a programmable network interface, using programming languages ​​such as P4, often allows for flexibility to better accommodate newer versions of multiple protocols used and to better adapt to the evolving needs of applications running on nodes in today's data centers.

[0083] Other implementation plans

[0084] Those skilled in the art will recognize that the various exemplary logic blocks, modules, and processes described in conjunction with the examples disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. Furthermore, the aforementioned processes can be embodied in a computer-readable medium that enables processor or controller circuitry to perform or implement certain functions.

[0085] To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, and modules have been generally described above in terms of their functionality. Whether this functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this disclosure.

[0086] The various exemplary logic blocks, cells, modules, processor circuits, and controller circuits described in conjunction with the examples disclosed herein can be implemented or executed using a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor can be a microprocessor, but alternatively, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor or controller circuitry can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, a SoC, one or more microprocessors incorporating a DSP core, or any other such configuration.

[0087] The activities of the methods or processes described in conjunction with the examples disclosed herein may be directly embodied in hardware, in a software module executed by processor or controller circuitry, or a combination of both. The steps of the method or algorithm may also be performed in an order alternative to the order provided in the examples. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable media, optical media, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor or controller circuitry such that the processor or controller circuitry can read information from and write information to the storage medium. Alternatively, the storage medium may be an integral part of the processor or controller circuitry. The processor or controller circuitry and the storage medium may reside in an ASIC or SoC.

[0088] The foregoing description of exemplary embodiments of this disclosure is provided to enable any person skilled in the art to make or use embodiments of this disclosure. Various modifications to these examples will be apparent to those skilled in the art, and the principles disclosed herein may be applied to other examples without departing from the spirit or scope of this disclosure. The embodiments described are to be considered merely exemplary and not restrictive in all respects. Furthermore, the language used in the following claims in the form of "at least one of A and B" should be understood to mean "only A, only B, or both A and B".

Claims

1. A programmable network interface for a server, the programmable network interface comprising: At least one memory, the at least one memory being configured to store connection parameters for multiple previously active network-based non-volatile memory Express (NVMeoF) connections to different non-volatile memory Express (NVMe) nodes; and The circuit is configured to: Receive NVMeoF connection requests from the NVMe node; Determine whether the NVMe node is associated with connection parameters stored in the at least one memory; as well as In response to determining that the NVMe node is associated with connection parameters stored in the at least one memory, a new NVMeoF connection is established for communicating with the NVMe node using the stored connection parameters.

2. The programmable network interface of claim 1, wherein the stored connection parameters include at least one of a connection status indication, negotiation parameters between the server and the NVMe node during a previous active connection, a packet sequence number, and a retransmission timeout value.

3. The programmable network interface of claim 1, wherein the circuitry is further configured to temporarily store data sent to the NVMe node after data is sent as a message.

4. The programmable network interface of claim 1, wherein the circuitry is further configured to: Send messages to the NVMe node through the network structure; Determine whether the timeout value stored in the connection parameters for receiving acknowledgments of sent messages from the NVMe node has been reached; and In response to the timeout value being reached without receiving the acknowledgment from the NVMe node, the message is resent to the NVMe node via the network structure.

5. The programmable network interface of claim 1, wherein the circuitry is further configured to: Receive an NVMeoF disconnect request from the NVMe node or from an application executed by the server; and In response to the received NVMeoF disconnect request, perform NVMeoF / TCP connection teardown on the new connection.

6. The programmable network interface of claim 1, wherein the circuitry is further configured to perform connection establishment with the NVMe node's NVMeoF / Transmission Control Protocol (TCP) connection, the connection establishment comprising: Based on the command received from the processor of the server, an NVMe / Transmission Control Protocol (TCP) initialization request is sent to the NVMe node; Receive NVMe / TCP initialization response from the NVMe node; Send an NVMeoF connection request to the NVMe node; as well as Receive the NVMeoF connection response from the NVMe node.

7. The programmable network interface of claim 1, wherein the circuitry is further configured to: Parse the Ethernet header of the message received from the NVMe node through the network structure; Parsing the NVMe capsule containing the message received from the NVMe node; and A portion of the NVMe capsule is provided to the server's processor, without providing the parsed Ethernet header and the remainder of the NVMe capsule.

8. The programmable network interface of claim 1, wherein the circuitry is further configured to: The data received from the server's processor is fragmented to fit the network's maximum transmission unit (MTU); and Multiple messages are sent to the NVMe node for the fragmented data.

9. The programmable network interface of claim 1, wherein the circuitry is further configured to: The address space of the server is partitioned; Allocate an NVMe request queue for each partition of the address space; Receive NVMeoF messages from the NVMe node; Identify at least one address in the received NVMeoF message; as well as Based on at least one identified address, an NVMe request queue is determined for executing NVMe requests included in the NVMeoF message, wherein the NVMe requests are executed by one or more processors of the server.

10. The programmable network interface of claim 1, wherein the circuitry is further configured to send an indication to the NVMe node of a predetermined number of credits for sending a message to the server based on the available capacity of the determined NVMe request queue, the predetermined number of credits being reduced by the NVMe node as each NVMeoF message is sent to the server.

11. A programmable network interface, the programmable network interface comprising: At least one memory, the at least one memory being configured to store connection parameters for multiple previously active network-based non-volatile memory Express (NVMeoF) connections to different non-volatile memory Express (NVMe) nodes; and Apparatus for the following operations: Send NVMeoF messages to NVMe nodes; Determine whether the timeout value stored in the connection parameters for receiving acknowledgments of the sent messages from the NVMe node has been reached; as well as In response to the timeout value being reached without receiving the acknowledgment from the NVMe node, the message is resent to the NVMe node via the network structure.

12. The programmable network interface of claim 11, wherein the programmable network interface further comprises means for performing the following operations: After sending data as a message, the data sent to the NVMe node is temporarily stored in the at least one memory; and In response to the timeout value being reached without receiving the acknowledgment from the NVMe node, the message is resent to the NVMe node via the network structure using temporarily stored data.

13. A method executed by a programmable network interface of a server, the method comprising: Receive non-volatile memory Express (NVMeoF) connection requests from NVMe nodes based on network architecture; Determine whether the NVMe node is associated with connection parameters stored in at least one memory of the programmable network interface, wherein the connection parameters are used for multiple previously active NVMeoF connections of different NVMe nodes; as well as In response to determining that the NVMe node is associated with connection parameters stored in the at least one memory, a new NVMeoF connection is established for communicating with the NVMe node using the stored connection parameters associated with the NVMe node.

14. The method of claim 13, wherein the stored connection parameters include at least one of a connection status indication, negotiation parameters between the server and the NVMe node during a previous active connection, a packet sequence number, and a retransmission timeout value.

15. The method of claim 13, further comprising temporarily storing the data sent to the NVMe node after the data is sent as a message.

16. The method of claim 13, further comprising: Send messages to the NVMe node through the network structure; Determine whether the timeout value stored in the connection parameters for receiving acknowledgments of the sent messages from the NVMe node has been reached; as well as In response to the timeout value being reached without receiving the acknowledgment from the NVMe node, the message is resent to the NVMe node via the network structure.

17. The method of claim 13, further comprising: Receive an NVMeoF disconnect request from the NVMe node or from an application executed by the server; as well as In response to a received NVMeoF disconnect request, perform NVMeoF / Transmission Control Protocol (TCP) connection teardown on the new connection.

18. The method of claim 13, further comprising performing connection establishment with the NVMeoF / TCP connection to the NVMe node, the connection establishment comprising: An NVMe / TCP initialization request is sent to the NVMe node based on a command received from the server's processor; Receive NVMe / TCP initialization response from the NVMe node; Send an NVMeoF connection request to the NVMe node; as well as Receive the NVMeoF connection response from the NVMe node.

19. The method of claim 13, further comprising: Parse the Ethernet header of the message received from the NVMe node through the network structure; Parse the NVMe capsule of the message received from the NVMe node; as well as A portion of the NVMe capsule is provided to the server's processor, without providing the parsed Ethernet header and the remainder of the NVMe capsule.

20. The method of claim 13, further comprising: The data received from the server's processor is fragmented to fit the network's maximum transmission unit (MTU); as well as Multiple messages are sent to the NVMe node for the fragmented data.

21. The method of claim 13, further comprising: The address space of the server is partitioned; Allocate an NVMe request queue for each partition of the address space; Receive NVMeoF messages from the NVMe node; Identify at least one address in the received NVMeoF message; as well as Based on at least one identified address, an NVMe request queue is determined for executing NVMe requests included in the NVMeoF message, wherein the NVMe requests are executed by one or more processors of the server.

Images