Merge packets based on hints generated by the network adapter

Abstract

The network node includes a network adapter and a host. The network adapter is coupled to a communication network. The host includes a processor running a client process and a communication stack, and is configured to receive packets from the communication network, and classify the received packets into respective flows associated with respective blocks in a receive buffer, distribute the payload of the received packets between the blocks, store the payload of the packets classified into a given flow in a given block assigned to the given flow, and notify the communication stack of the payload in the given block, so that the payload in the given block is transmitted to the client process.

Description

Technical Field

[0001] Implementations described herein relate generally to data communications, and more particularly to methods and systems for merging packets based on hints generated by network adapters. Background Art

[0002] A client process running on a host computer can access a communication network using a suitable communication protocol. For example, a client process can access the transport layer of a communication protocol (e.g., Transmission Control Protocol (TCP)) using a software interface called a "socket".

[0003] Methods for interfacing transport layers using sockets are known in the art. For example, U.S. Patent Application Publication 2014 / 0229946 describes systems and methods for implementing network communication I / O operations for virtual machines. An example method may include: identifying a network packet sent to a virtual process controlled by a client operating system, identifying a memory location corresponding to a virtual memory location used by the virtual process, and copying data of the network packet to a memory location corresponding to the virtual memory location. Another example method may include: retrieving data to be included in the network packet from a first memory location, wherein the first memory location corresponds to a virtual memory location used by a virtual process controlled by the client operating system, retrieving a header included in the network packet from a second memory location, wherein the second memory location corresponds to a kernel buffer location of the client operating system, and combining the header and data to form a network packet. Summary of the invention

[0004] The embodiments of the present invention described herein provide a network node, including a network adapter and a host. The network adapter is coupled to a communication network. The host includes a processor running a client process and a communication stack, and is configured to receive packets from the communication network, and classify the received packets into corresponding flows associated with corresponding blocks in a receive buffer, distribute the payload of the received packets between the blocks, so that the payload of the packets classified into a given flow is stored in a given block allocated to the given flow, and notify the communication stack of the payload in the given block to transmit the payload in the given block to the client process.

[0005] In some embodiments, the processor is further configured to run a driver that mediates between the network adapter and the communication stack, wherein the network adapter is configured to notify the communication stack of the payload in the given block via the driver. In other embodiments, the driver is configured to construct a merged payload including two or more consecutive payloads in the given block and notify the communication stack of the merged payload. In other embodiments, the payload in the given block includes a first payload and a second payload that meet a matching criterion for merging, and wherein the driver is configured to avoid merging the first payload and the second payload based on criteria other than the matching criterion.

[0006] In one embodiment, the first payload and the second payload in the given block belong to packets of different corresponding streams, and wherein the network adapter is configured to notify the driver that the first payload and the second payload do not match for merging. In another embodiment, in response to detecting that the storage space available in the given block is less than the payload to be stored in the given block, the network adapter is configured to allocate another block to the given stream and store the payload in the another block. In yet another embodiment, in response to detecting that the storage space available in the given block is less than the payload to be stored in the given block, the network adapter is configured to allocate another block to the given stream and split storage of the payload between the block and the another block.

[0007] In some embodiments, the received packets belong to at least a first stream and a second stream, and wherein the network adapter is configured to allocate different corresponding blocks in the receive buffer to the first stream and the second stream. In other embodiments, the communication stack is configured to apply a direct data transfer of two or more payloads stored consecutively in the group blocks to user space. In other embodiments, the receive buffer resides in a memory of the host, wherein the communication stack is configured to apply the direct data transfer only when the two or more payloads (i) are aligned to the operating system page in the receive buffer and (ii) have an operating system page granularity.

[0008] In one embodiment, the communication stack is included in a kernel of an operating system running in kernel space, and wherein the communication stack is configured to transmit one or more payloads in the given block to the client process in user space. In another embodiment, the communication stack includes a communication program running in user space, and wherein the communication stack is configured to transmit one or more payloads in the given block directly to the client process in user space. In yet another embodiment, the network adapter is configured to store the header of the received packet in a header buffer and notify the communication stack of the stored header corresponding to the payload of the received packet in the given block. In some embodiments, the communication stack or a driver that mediates between the network adapter and the communication stack is configured to generate a merged packet to be transmitted to the client process based on the header stored in the header buffer and the one or more payloads stored in the given block.

[0009] According to the embodiments described herein, there is further provided a communication method, including in a network node including a network adapter coupled to a communication network, the network node including a host, the host including a processor running a client process and a communication stack, receiving packets from the communication network by the network adapter, and classifying the received packets into corresponding flows associated with corresponding blocks in a receive buffer. Distributing the payload of the received packets between the blocks so that the payload of the packets classified into a given flow is stored in a given block assigned to the given flow. Notifying the communication stack of the payload in the given block so that the payload in the given block is transmitted to the client process.

[0010] According to the embodiments described herein, a network adapter is provided, which includes a network interface, a host interface and a packet processing circuit. The network interface is configured to connect to a communication network. The host interface is coupled to a host, which includes a processor running a client process and a communication stack. The packet processing circuit is configured to receive packets from the communication network, and classify the received packets into corresponding flows associated with corresponding blocks in a receive buffer, distribute the payload of the received packets between the blocks, so that the packets classified into a given flow are stored in a given block assigned to the given flow, and notify the communication stack of the payload in the given block to transmit the payload in the given block to the client process.

[0011] According to the embodiments described herein, there is further provided a method for communication, including in a network adapter connected to a communication network and a host, the host including a processor running a client process and a communication stack, receiving packets from the communication network, and classifying the received packets into corresponding flows associated with corresponding blocks in a receive buffer. Distributing the payload of the received packets between the blocks so that packets classified into a given flow are stored in a given block assigned to the given flow. Notifying the communication stack of the payload in the given block so that the payload in the given block is transmitted to the client process.

[0012] These and other embodiments will be more fully understood from the following detailed description of embodiments thereof, taken in conjunction with the accompanying drawings, in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram schematically illustrating a computing system according to embodiments described herein, wherein a network adapter provides hints to a host for merging received packets;

[0014] Figure 2 is a diagram schematically illustrating a receive buffer divided into a plurality of blocks for storing the payload of a received packet according to an embodiment described herein;

[0015] Figure 3 is a diagram schematically illustrating a scheme for buffering received packets belonging to a common flow according to embodiments described herein;

[0016] Figure 4 is a diagram schematically illustrating a scheme for buffering received packets belonging to multiple flows according to embodiments described herein; and

[0017] Figure 5 is a flow chart schematically illustrating a method for merging packets into blocks according to an embodiment described herein. DETAILED DESCRIPTION

[0018] Overview

[0019] Implementations described herein provide systems and methods for merging packets based on hints generated by network adapters.

[0020] A network node typically includes a host coupled to a communication network using a network adapter or a network interface controller (NIC). The host includes a processor such as a central processing unit (CPU) and a memory. The network adapter receives packets from the communication network and forwards the received packets to an upper software layer. The CPU typically runs a communication stack, a driver that mediates between the network adapter and the communication stack, and a client process such as a user application.

[0021] The communication stack may be included in the kernel program of the underlying operating system (OS) (e.g., Linux OS). In an alternative embodiment, a data plane development kit (DPDK) may also be used. DPDK runs in user space and provides open source libraries and network adapter drivers for offloading packet processing from the operating system kernel. The DPDK project is managed by the Linux Foundation. Further alternatively, any other suitable communication stack may also be used.

[0022] In the following description, it is assumed that user applications, client processes, and DPDK can access an area in memory referred to herein as "user space", and kernel programs can access a different area in memory, referred to herein as "kernel space". In kernel-based communications, packets received from a communication network via a network adapter are typically first buffered in kernel space and then transferred to user space by the kernel program. In construction, DPDK bypasses the kernel program and transfers the received data directly to user space.

[0023] In kernel-based communication, to receive data from a communication network, a client process provides the kernel with a socket buffer and the amount of data to be consumed. The network adapter receives packets from the communication network destined for the client process and stores them in a socket buffer. The kernel then copies the data from the socket buffer (or a buffer pointed to from the socket buffer) to user space memory accessible to the client process. The copy operation to user space can be costly in terms of CPU cycles, communication throughput, and latency.

[0024] Packets received from a communication network usually have a limited size. A size limit called the Maximum Transmission Unit (MTU) defines the size of the largest packet that can be sent over the wire in the underlying network, for example 1.5KB. Processing small packets requires a lot of processing, resulting in high latency and reduced throughput because the header of the packet is processed separately.

[0025] In principle, to improve performance, multiple packets of a common data flow can be merged to produce larger packets for processing by higher layers of the network stack, for example, as in the Large Receive Offload (LRO) approach. The main disadvantage of LRO and similar approaches is that the communication stack only receives the merged large packets and has no flexibility in controlling the packet merging process.

[0026] In a disclosed embodiment, the network adapter stores the header and payload of each received packet separately. The network adapter reports to the communication stack (e.g., via a driver) the payloads and corresponding headers of multiple packets belonging to a common flow. The communication stack or driver may combine some or all of the reported payloads to create at least one merged packet including two or more reported payloads, wherein the header of the merged packet is based on the report header.

[0027] The Linux 4.18 version in August 2018 introduced the receive zero copy function. The Linux function only supports zero copying of continuous memory areas that are aligned with the operating system page and whose size is equal to an integer multiple of the operating system page size (e.g., 4KB). The Linux zero copy function applies memory address mapping technology. In some embodiments, the zero copy function is implemented in an underlying communication protocol such as the Transmission Control Protocol (TCP).

[0028] For example, a kernel zero-copy transfer can be invoked by transmitting packets with a payload size equal to the operating system page size. However, such a configuration is limited and does not include important use cases, such as using an operating system page size of 4KB and a maximum transmission unit (MTU) of 1.5KB. The MTU is typically configured to be 1.5KB, which makes it impossible to send packets with an operating system page size (e.g., 4KB). One disadvantage of using a relatively small MTU is that the source needs to break up large amounts of data into multiple smaller units, each of which is sent in a separate packet, to meet the MTU constraint. This consumes network bandwidth and processing resources due to the multiple packet headers.

[0029] In principle, the MTU value could be increased. However, this is often not possible in many practical use cases. Furthermore, even if the MTU could be configured to be around 4KB, using the kernel zero-copy feature would require that only the payload of the packet be spread across the socket buffer (excluding the header), aligned to the operating system page.

[0030] In some disclosed embodiments, a communication stack is included in a kernel program that applies zero-copy data transfers to user space only for data that is aligned to an operating system page in memory and has page granularity. A network adapter applies a novel buffering scheme to incoming packets to force the kernel zero-copy function to be invoked.

[0031] In some embodiments, the network node includes a network adapter and a host. The network adapter is coupled to a communication network. The host includes a processor and a communication stack running a client process. The network adapter classifies received packets into corresponding flows associated with corresponding blocks in a receive buffer. The network adapter distributes the payload of the received packets into blocks so that packets classified into a given flow are stored in a given block assigned to the given flow, and notifies the communication stack of the payload in the given block to transmit the merged packets to the client process.

[0032] In this context, a "flow" comprises a sequence of packets originating from a common source and destined for a common destination. A flow can be identified by one or more predefined fields in the packet header.

[0033] In some embodiments, the processor is further configured to run a driver that mediates between the network adapter and the communication stack, and the network adapter is configured to notify the communication stack of the payload in the given block via the driver. The driver constructs a merged payload that includes two or more consecutive payloads in the given block and notifies the communication stack of the merged payload. In one embodiment, the payload in the given block includes a first payload and a second payload that meet a matching criterion for merging, but the driver avoids merging the first payload and the second payload based on criteria other than the matching criterion.

[0034] In an embodiment, a first payload and a second payload in a given block belong to packets of different corresponding flows. This may occur when multiple flow identifiers are hashed into a common hash result. In this case, the network adapter notifies the driver that the first payload and the second payload do not match for merging. The network adapter additionally stores the header of the received packet in a header buffer and notifies the communication stack of the stored header corresponding to the payload of the packet received in the given block. The communication stack, or a driver that mediates between the network adapter and the communication stack, generates a merged packet to be transmitted to the client process based on the header stored in the header buffer and the one or more payloads stored in the given block.

[0035] Each block has limited storage space in the receive buffer. In response to detecting that the storage space available in a given block is less than the payload to be stored in the given block, the network adapter allocates another block for the given flow and stores the payload in the other block. Alternatively, the network adapter splits the storage payload between the block and the other blocks.

[0036] In some implementations, the received packets belong to at least a first flow and a second flow, and the network adapter allocates different respective blocks in the receive buffer to the first flow and the second flow.

[0037] In an embodiment, the communication stack applies a direct data transfer of two or more payloads stored consecutively in a block to the user space. When the receive buffer resides in the host's memory, the communication stack may apply the direct data transfer (e.g., a zero-copy transfer) only when the two or more payloads (i) are aligned with an operating system page in the receive buffer and (ii) have an operating system page granularity.

[0038] In an embodiment, the communication stack is included in a kernel of an operating system running in kernel space, and the communication stack is configured to transmit one or more payloads in a given block to a client process in user space. In another embodiment, the communication stack includes a communication program running in user space, and the communication stack is configured to transmit one or more payloads in a given block directly to a client process in user space.

[0039] In the disclosed technology, the network adapter stores the received payloads of the packets belonging to the common flow in the same block. The network adapter reports the matching hint between the payload in the block and the continuous payload to the communication stack, for example, via the driver. The communication stack or the driver can flexibly construct one or more merged packets based on the reported payload. Processing the merged packets including the corresponding merged payloads is generally more efficient than processing a single packet, so by using the disclosed embodiments, the communication throughput is increased.

[0040] System Description

[0041] Figure 1 is a block diagram schematically illustrating a computing system 20 in which a network adapter provides hints to a host for coalescing received packets, according to embodiments described herein.

[0042] In computing system 20, network node 24 communicates with network node 28 via communication network 30. Communication network 30 may include any suitable packet network, such as Ethernet or an IP network, such as the Internet. Communication network 30 may operate using any suitable communication protocol, such as Internet Protocol (IP), Transmission Control Protocol (TCP), or both. In another embodiment, the communication network includes an infinite broadband TM (IB) architecture, in which case the packets are transported using TCP and IP over IB (IPoIB) protocols.

[0043] The computing system 20 may be used in any application where network nodes exchange large amounts of data, such as high performance computing (HPC), distributed computing, and data centers.

[0044] The network node 24 includes a processor in the form of a central processing unit (CPU) 32 and a memory 34, which are interconnected by a suitable bus 36. In the present example, the bus 36 comprises a peripheral component interconnect express (PCIe) bus. The network node 24 also includes a network adapter 38, such as a network interface card (NIC), for connecting to the communication network 30. The combination of the CPU 32, the memory 34 and the bus 36 (with or without a network adapter) is also referred to as a "host computer" or simply a "host" for brevity. The memory 34 is also referred to as "host memory" or "system memory".

[0045] The network adapter 38 includes a network interface 42 coupled to the communication network 30 and a host interface 44 coupled to the CPU 32 and the memory 34 via the bus 36. A packet processing circuit 46 coupled between the network interface 42 and the host interface 44 generates outgoing packets for transmission on the communication network 30 and processes incoming packets received from the communication network, as described below. In the ingress direction, the packet processing circuit 46 applies various processing to packets received in the network element via the network interface 42, such as verifying the correctness of the data in the packet payload, packet classification and prioritization, and routing. The packet processing circuit typically checks certain fields in the header of the packet for packet classification and routing. The header fields contain addressing information, such as source and destination addresses and port numbers, as well as the underlying network protocol used.

[0046] The network interface 42, the host interface 44 and the packet processing circuit 46 typically comprise dedicated hardware logic. Alternatively or additionally, at least some of the functionality of the packet processing circuit 46 may be implemented in software on a suitable programmable processor.

[0047] The CPU 32 runs a communication program 50 that handles host communications on the network communication 30 via the network adapter 38. The communication program 50 is also referred to as a "communication stack" in this article. The communication stack 50 can be included in, for example, a kernel program that is part of an operating system (OS) such as "Linux" (not shown). Alternatively, the communication stack bypasses the kernel program, such as a data plane development kit (DPDK). DPDK is open source software managed by the Linux Foundation. Compared with the Linux kernel driven by the interrupt generated by the network adapter, DPDK achieves higher packet throughput.

[0048] The communication stack 50 includes a communication protocol including a transport layer 52, such as TCP. A driver 54 runs below the transport layer 52 and mediates between the communication stack 50 and the network adapter 38.

[0049] A client process 60 (also referred to simply as “client” for brevity), such as a process generated by application software, running on CPU 32 communicates with a peer client (not shown) running on remote network node 28 .

[0050] As will be described in detail below, when receiving packets from the communication network 30, the driver 54 manages the allocation of resources such as buffers and queues, which will be used by the network adapter 38 for packet reception. In some embodiments, the network adapter 38 stores the payload of multiple packets in the receive buffer 64, aligned with the operating system page. The driver 54 monitors the packet reception in the receive buffer 64 and initiates the data transmission from the receive buffer to the user buffer 72 in the user space accessible to the relevant client process, for example, via the communication buffer 68. The communication buffer 68 may include, for example, a TCP socket buffer (SKB). In some embodiments, as will be described in detail below, the communication stack supports zero-copy data transmission of buffered data stored with page alignment and page granularity to the user space with the operating system page.

[0051] Figure 1 The computing system, network node, and network adapter configurations shown in are example configurations, and these configurations are selected purely for conceptual clarity. In alternative embodiments, any other suitable computing system, network node, and network adapter configuration may also be used. For clarity, components that are not necessary for understanding the principles of the present invention, such as various interfaces, addressing circuits, timing and sorting circuits, and debugging circuits, are omitted in the figure.

[0052] Some elements of network adapter 38 (e.g., packet processing 46) may be implemented in hardware, for example, in one or more application specific integrated circuits (ASICs) or FPGAs. Additionally or alternatively, packet processing circuitry 46 may be implemented using software or using a combination of hardware and software elements. Memory 34 may include any suitable type of memory using any suitable storage technology, such as random access memory (RAM), dynamic RAM (DRAM), non-volatile memory such as flash memory, or a combination of multiple memory types.

[0053] In some embodiments, some functions of the network adapter 38 may be performed by a general purpose processor, such as the packet processing circuit 46, which is programmed in software to perform the functions described herein. The software may be downloaded to the processor in electronic form, such as over a network, or it may alternatively or additionally be provided and / or stored on a non-transitory tangible medium, such as a magnetic, optical or electronic memory.

[0054] Receive buffer management

[0055] Figure 2is a diagram schematically illustrating a receive buffer divided into a plurality of chunks for storing payloads of receive packets according to embodiments described herein.

[0056] exist Figure 2 In the embodiment of the present invention, a circular shared receive queue (SRQ) 76 includes a plurality of work queue elements (WQEs) 78, each of which includes a pointer to a corresponding receive buffer 64. In some embodiments, the WQE stores information about block usage, such as the occupancy of the currently used block and the next available block in the receive buffer.

[0057] In some embodiments, the driver 54 creates an SRQ 76 upon initialization, such as after a reset, and posts a plurality of WQEs 78 to the SRQ before starting packet reception. The network adapter then executes the posted WQEs in sequence (and in a loop). When the currently used receive buffer is full, the network adapter switches to a subsequent WQE pointing to an idle receive buffer. In some embodiments, the network adapter 38 automatically reposts the consumed WQEs to the SRQ 76, thereby freeing the client from participating in posting WQEs. In some embodiments, the network adapter 38 can add WQEs to or remove WQEs from the SRQ 76 as needed.

[0058] In some embodiments, the receive buffer 64 includes (or is divided into) a plurality of blocks 80, wherein each block includes a continuous address range in the memory 34. In some embodiments, the block 80 is aligned with an operating system page, which means that the starting address of the block 80 in the memory 34 is aligned with the address of the corresponding operating system page. In the following description, for the sake of brevity, "operating system page" is referred to as "page".

[0059] In this context, a region in memory 34 whose start address is aligned with a corresponding page of the operating system is referred to herein as "page aligned." When the size of a page aligned region is an integer multiple of the operating system page size, the region is also referred to as having "page granularity."

[0060] In some implementations, block 80 is page aligned and has page granularity. Figure 2 In the example of , the receive buffer has a continuous address range, which is divided into multiple blocks. In the example implementation, the operating system page size is 4KB, the block size is 64KB (16 pages), and the receive buffer size is 1MB (16 blocks).

[0061] When the block 80 is page-aligned, a contiguous region of addresses starting from the beginning of the block and having page granularity will be zero-copied to user space by the communication stack 50 (e.g., when zero copy is supported and enabled). On the other hand, when the size of the contiguous region is not an integer multiple of the operating system page size, the first portion of the region with page granularity will be zero-copied, while the remainder of the region (less than the page size) will be conventionally copied to user space. Therefore, in order to increase the amount of data that is zero-copied, the block 80 should be configured to be page-aligned and have page granularity.

[0062] As will be described below, network adapter 38 stores only the payload of the incoming packet (and separately the header of the incoming packet) in block 80. In the disclosed embodiments, any suitable payload size may be used, and different packets may have payloads of the same or different sizes.

[0063] Although in Figure 2 All blocks 80 in the datagram have a common size, but this configuration is not mandatory. In alternative embodiments, blocks of different sizes can also be used. For example, the block size can depend on the data rate and / or the priority of the stream.

[0064] Group buffering and merging methods

[0065] Figure 3 is a diagram schematically illustrating a scheme for buffering received packets belonging to a common flow according to embodiments described herein.

[0066] exist Figure 3 , the network adapter 38 receives packets 82 from the communication network 30, each packet including a header 83 and a payload 84. The network adapter 38 processes the packets 82 and separates each packet into its header and payload for separate storage. Specifically, the network adapter scatters the headers 83 (e.g., sequentially) in a dedicated header buffer 86, and continuously scatters the payloads 84 of the same stream in blocks 80 of the receive buffer 64. The network adapter stores the first payload at the beginning of the block, stores the second payload immediately after the first payload, and so on. The payloads scattered in the same block form a large combined payload 92.

[0067] In some embodiments, the network adapter selects one of the headers 83 in the header buffer 86 as the header of the large packet including the merged payload 92, and modifies the selected header accordingly. Alternatively, the network adapter generates a header for the merged payload 92 independently of the header 83. By merging multiple payloads in a block, a large amount of data can be efficiently transferred to user space by the communication stack, or zero-copied in a single data transfer operation. In addition, the underlying transport layer 52 (e.g., TCP) receives large packets instead of multiple small packets, thereby reducing the overhead generated by transmitting and processing all packet headers.

[0068] Note that in some applications, to send a large message over a communications network, the source of the packet needs to break the message into multiple payloads for transmission within multiple packets to meet the MTU limit of the network. Using the above-described coalescing mechanism, the network adapter at the receiving end reproduces the large message (or a portion thereof) by storing the related payloads consecutively in blocks, as described above.

[0069] For each received packet 82, the network adapter 38 identifies the associated receive buffer to which the current WQE 78 is pointing. As will be described below, the network adapter spreads the packet payload among the blocks of the receive buffer that have been allocated to the flow to which the packet belongs.

[0070] In some embodiments, the network adapter 38 generates a completion queue element (CQE) 88 for each packet 82 that has been separated into separate headers and payloads and buffered as described above, and writes the CQE to a completion queue (CQ) 90. For a given packet 82, the CQE 88 may store information such as a block identifier, the starting address of each payload within the block, the size of the stored payload, etc. As will be described in more detail below, the CQE may also store an indication of whether the payload matches a previous payload in the block for merging using appropriate matching criteria. The information in the CQE provides a hint to the driver 54 to decide to merge the payloads in the block into one or more merged payloads (92).

[0071] In some embodiments, the driver 54 monitors the CQE 88 to decide to close the current block and initiate the transfer of the block contents to user space. In some embodiments, when the driver 54 identifies that the current block is full or nearly full, the driver 54 allocates a socket buffer (SKB) 68 pointing to (i) the merged payload 92 and (ii) the modified header 83. The communication stack 50 then transfers the merged payload 92 to user space.

[0072] When the communication stack supports zero-copy data transfer and zero-copy is enabled, the communication stack transfers the merged payload (or a portion thereof with page granularity) to user space. In the case where the size of the merged payload 92 is not an integer multiple of the page size, a portion of the merged payload with page granularity will be zero-copied to user space, and the remainder of the merged payload will be conventionally copied to user space.

[0073] Figure 4 is a diagram schematically illustrating a scheme for buffering received packets belonging to multiple flows according to embodiments described herein.

[0074] exist Figure 4In the example of FIG. 4 , the network adapter 38 receives packets belonging to a plurality of (three in this example) flows represented as Flow 1, Flow 2, and Flow 3 from the communication network 30 via the network interface 42. In actual applications, depending on the underlying application, the packets may belong to only a few concurrent flows or may belong to a greater number of concurrent flows, such as hundreds or thousands of flows.

[0075] exist Figure 4 In , the received packets are represented as packets (i, j), where 'i' represents a flow identifier, and j = 0, 1, 2, ... represents the sequence number of the packets in the i-th flow. In this example, the five packets received by the network adapter 38 include packets (1, 0) and (1, 1) belonging to flow 1, packets (2, 0) and (2, 1) belonging to flow 2, and packets (3, 0) belonging to flow 3. Figure 4 In , packets are also referenced by an index in the range 1…5, which indicates the order in which the packets were received.

[0076] In some implementations, the currently used WQE 78 maintains a pointer to the receive buffer 64 and another pointer to the header buffer 86. These pointers may be provided by the driver 54 to the network adapter.

[0077] In some embodiments, the driver 54 allocates a corresponding receive queue (RQ) 94 for each received stream. The RQ is associated with a corresponding stream context that holds the stream identifier of the stream served by the RQ. At any given time, the network adapter allocates a corresponding block 80 in the receive buffer 64 to each RQ (or the stream served by the RQ). The stream context of the RQ holds the block identifier of the current block. The CQE associated with the block holds information about the address and size of the payload that has been stored in the current block, as well as matching information for merging between consecutive payload pairs in the block.

[0078] The network adapter 38 classifies the received packets into the corresponding flows to which they belong and directs the packets to the relevant RQs servicing those flows. The network adapter stores the packet headers in the header buffer 86 and stores the packet payloads in the currently used blocks. In some embodiments, the network adapter stores the payloads of packets belonging to a common flow consecutively in the same block. When the current block is closed and a new block is assigned to a flow, the network adapter begins to store the payloads of subsequent packets sequentially and consecutively in the new block. Figure 4 In, as mentioned above Figure 3 As described above, the payloads of the same stream are merged into a large payload.

[0079] Although for clarity Figure 4 , but the network adapter writes a CQE indicating the stored payload, as referenced above Figure 3 described.

[0080] exist Figure 4 , the network adapter stores the payloads of packets belonging to stream 1, stream 2, and stream 3 in separate corresponding blocks, as shown below. In the block corresponding to stream 1, the network stores the first and fourth packets received. In the block corresponding to stream 2, the network adapter stores the second and fifth packets received. In the block corresponding to stream 3, the network adapter stores the third packet received.

[0081] Figure 5 is a flow chart schematically illustrating a method for merging packets into blocks according to embodiments described herein.

[0082] In describing the method, it is assumed that the network adapter receives packets belonging to one or more flows serviced by corresponding RQs. In addition, each RQ is allocated a corresponding block in the common receive buffer 64. It is assumed that the blocks are page-aligned and have a page granularity.

[0083] The method will be described as being performed by network adapter 38 (or more specifically, packet processing circuitry 46).

[0084] The method begins with the network adapter 38 receiving a packet from the communication network 30 via the network interface 42 at a packet receiving step 100. The packet may include, for example, a TCP packet or a TCP / IP packet. In another embodiment, instead of TCP, a user datagram protocol (UDP) may also be used. At an RQ determination step 104, the network adapter directs the packet to the RQ of the stream to which the service received packet belongs. In some embodiments, the network adapter determines the RQ by applying a hash function to one or more fields in the packet header. The hash function may include, for example, a Toeplitz hash function. For example, the network adapter applies a hash function to a five-tuple in a header including (protocol, source address, source port, destination address, destination port).

[0085] In a block identification step 108, the network adapter extracts an identifier (e.g., a pointer or block index) for the current block of the stream served by the RQ from the RQ stream context. Further in step 108, the network adapter sets the gap variable to a zero value, which means that the payload of the received packet is stored immediately after the previously stored payload. In a match check step 112, the network adapter checks whether the current packet matches a previously stored packet for merging. The purpose of the match check is to identify a sequence of packets whose payloads can be merged (by the driver) into a larger packet. Checking for a match in step 112 can be performed in various ways. For example, a match occurs when the packet of the most recently stored payload and the currently received packet have consecutive sequence numbers.

[0086] Another matching criterion checks whether the flow served by the RQ (determined at step 104) matches (e.g., has the same flow identifier, such as a five-tuple value) the flow to which the received packet belongs. This matching criterion is necessary because hash functions may be imperfect and result in hash collisions, which means that five-tuples (or other predefined fields in the packet header) that hash differently may result in the same hash result and therefore have the same RQ. Hash collisions may also occur when the number of RQs is less than the number of supported flows.

[0087] When a match occurs at step 112, the network adapter proceeds to a match indication step 114, where the network adapter sets the CQE indicator variable to "match". At a block occupancy check step 116, the network adapter checks whether the remaining storage space in the current block is large enough to store the payload of the received packet (possibly including a non-zero gap). If so, and if the gap value is zero, the network adapter 38 spreads the entire payload of the received packet in the current block contiguously to the already stored payload (if any) at a complete payload storage step 120.

[0088] At a header storage step 124, the network adapter scatters the header of the received packet in the header buffer 86. At a completion step 128, the network adapter writes the CQE to the CQ 90. In an embodiment, the CQE stores a state or state value indicating whether a match occurred at step 112 above. At step 128, the network adapter sets the state value in the CQE to the current value of the CQE indicator variable. After step 128, the method loops back to step 100 to receive subsequent packets.

[0089] At step 116, when the remaining storage space in the block is less than the payload size, in one embodiment, network adapter 38 may spread the entire payload to the new block, or alternatively, split the payload between the current block and the new block at steps 136, 140 and 144, as described herein.

[0090] In a block fill step 136, the network adapter continues to spread partial payloads into the current block until the entire block is filled (possibly after a non-zero gap). In a new block allocation step 140, the network adapter closes the current block and allocates a new block available in the receive buffer to the RQ.

[0091] The network adapter scatters the remaining portion of the payload that was not scattered into the enclosing block into the new block in the new block scatter step 144. After step 144, the method proceeds to step 124 to write the header of the received packet, and then writes the CQE (with the current CQE indicator value) in step 128 as described above.

[0092] When no match occurs in step 112, the network adapter prepares to store the packet payload continuously (gap value is zero) in the current group block or store a non-zero gap relative to the previously stored payload in the gap determination step 152. The gap size can be configured as, for example, one or more bytes, or a system page size. In some embodiments, the gap is selected so that the payload of the current packet will be stored aligned with an operating system page or a system cache line. When there is no room for the current payload in the current block, the network adapter closes the current block and assigns a new block in the receive buffer to the RQ. When the current block contains packets from two different streams, only the portions of the two streams that are aligned with the operating system page and have page granularity can be zero-copied to user space by the communication stack.

[0093] Note that the use of a non-zero gap is not mandatory. In alternative embodiments, the payload of the current packet can be stored contiguously with the previous payload of a different stream. With or without a gap, the CQE reports the start and end addresses of the payloads belonging to the different streams, so the two payloads are copied to the user space separately. In these embodiments, the communication stack applies the data transfer from the receive buffer to the user space. The hardware circuitry in the network adapter stores the payloads of the packets classified as a given stream contiguously in the block of the receive buffer allocated to the given stream so that the communication stack transfers the entire block of the contiguously stored payload to the user space.

[0094] At a mismatch indication step 156, the network adapter 38 sets the network adapter CQE indication variable to "mismatch". As described above, the "mismatch" indication will be reported in the CQE status value at step 128. After step 160, the method proceeds to step 116 and further steps to store the payload and header of the received packet in the current block or a new block.

[0095] Groups merged by the driver

[0096] In the LRO method mentioned above, the transport layer receives complete merged packets. In contrast, the driver 54 receives per-packet information about the payload, header, and matching hints of the received packets from the network adapter 38. Using such detailed information, the driver 54 has the flexibility to determine packet merging that would otherwise not be possible. For example, even if the network adapter notifies that the payloads match for merging, the driver may decide not to merge consecutive payloads in a block. Typically, the driver will not merge unmatched payloads, but is free to avoid merging matching payloads, for example, using matching criteria other than the one used by the network adapter.

[0097] In one embodiment, the driver determines the number of packets (or payloads) to merge based on the CPU load. For example, the driver may increase the number of merged packets as the CPU load increases. Because the network adapter merges packets in a "best effort" manner in the disclosed technology, the likelihood of having pending packets in the network adapter's receive buffer while the CPU is free to process them is significantly reduced. Note that the network adapter sends the processed and merged packets to the host immediately along with a completion notification without further storage in the network adapter.

[0098] In some embodiments, the communication stack is included in a Linux kernel that supports a new API (NAPI). A driver using NAPI provides a polling method to the kernel for obtaining multiple packets from a network adapter. During a NAPI polling cycle, the kernel thus receives multiple packets from the network adapter via the driver.

[0099] The above embodiments are given as examples, and other suitable embodiments may also be used.

[0100] Although the embodiments described herein primarily address storing the payload of packets received from the network, aligned with system pages to enforce the use of the Linux zero-copy feature, the methods and systems described herein may also be used for other applications, such as other applications that require fast transfer of data to user space where the underlying communication stack supports zero-copy data transfer.

[0101] In the above-described embodiments, the payloads of multiple packets are primarily merged to produce a large payload that is effectively zero-copied to user space when alignment and page granularity conditions are met, as described above. A header is associated with this large payload and provided to TCP as a corresponding large packet. In other embodiments, merging multiple payloads of received packets to produce a large packet that is effectively provided to TCP is independent of the alignment of the large payload in memory or whether the large payload is transferred to user space using standard copy or zero copy.

[0102] In the above-described embodiment, the receiver buffer for storing the payload of the incoming packets and the header buffer for storing the header of the incoming packets reside in the host. However, this is not mandatory. In alternative embodiments, the receive buffer and the header buffer may reside in the memory of the network adapter, in the graphics processing unit (GPU), or in any other element, so that the receiver buffer and the header buffer are accessible by the network adapter and the host.

[0103] It should be understood that the embodiments described above are cited by way of example, and the following claims are not limited to what has been specifically shown and described above. On the contrary, the scope includes combinations and sub-combinations of the various features described above, as well as changes and modifications that would occur to a person skilled in the art after reading the above description and that are not disclosed in the prior art. Documents incorporated by reference into this patent application should be considered an integral part of this application, unless any term defined in these incorporated documents in a manner that conflicts with an explicit or implicit definition in this specification, only the definition in this specification should be considered.

Claims

1. A network node, comprising: a network adapter coupled to a communications network; and A host computer, the host computer comprising a processor running a client process and a communication stack; Where the network adapter is configured as: receiving packets from the communication network and classifying the received packets into respective flows associated with respective blocks in a receive buffer; distributing the payloads of the received packets among the blocks so that the payloads of the received packets classified to a given flow are stored in a given block allocated to the given flow; as well as indicating to said host a packet of said received packets matching the merge, The host is configured to decide whether to merge packets indicated as matching for merging, and to generate a merged packet for the packets decided to be merged, the merged packet having a corresponding header of the merged packet.

2. The network node of claim 1, wherein when a first payload and a second payload in the given block belong to packets of different respective flows, the network adapter is configured to notify a driver that the first payload and the second payload are not compatible for merging.

3. The network node of claim 1 , wherein in response to detecting that the available storage space in the given block is less than the payload to be stored in the given block, the network adapter is configured to allocate another block to the given flow and store the payload in the another block.

4. The network node of claim 1 , wherein in response to detecting that the storage space available in the given block is less than the payload to be stored in the given block, the network adapter is configured to allocate another block to the given flow and split storage of the payload between the block and the another block.

5. The network node of claim 1, wherein the received packets belong to at least a first flow and a second flow, and wherein the network adapter is configured to allocate different respective blocks in the receive buffer to the first flow and the second flow.

6. The network node of claim 1, wherein the communication stack is included in a kernel of an operating system running in kernel space, and wherein the communication stack is configured to transmit one or more payloads in the given block to the client process in user space.

7. The network node of claim 1, wherein the communication stack comprises a communication program running in a user space, and wherein the communication stack is configured to transmit one or more payloads in the given block directly to the client process in the user space.

8. The network node of claim 1, wherein the network adapter is configured to store a header of the received packet in a header buffer and to notify the communication stack of the stored header corresponding to a payload of the received packet in the given block.

9. The network node of claim 8, wherein the communication stack or a driver mediating between the network adapter and the communication stack is configured to generate the merged packet based on the header stored in the header buffer and one or more payloads stored in the given block.

10. A communication method, comprising: In a network node including a network adapter coupled to a communications network, the network node including a host including a processor running a client process and a communications stack, receiving, by the network adapter, packets from the communications network and classifying the received packets into respective flows associated with respective blocks in a receive buffer; distributing the payloads of the received packets among the blocks so that the payloads of packets classified to a given flow are stored in a given block allocated to the given flow; indicating, by the network adapter, to the host, packets that match for merging; determining, by the host, whether to merge packets indicated as matching for merging; as well as The host generates a merged packet for the packets determined to be merged, the merged packet having a corresponding header of the merged packet.

11. The method of claim 10, wherein when the first payload and the second payload in the given block belong to packets of different respective flows, notifying the communication stack comprises notifying the communication stack via a driver that the first payload and the second payload are not compatible for merging.

12. The method of claim 10, wherein in response to detecting that the storage space available in the given block is less than the payload to be stored in the given block, allocating another block to the given stream and storing the payload in the another block.

13. The method of claim 10, wherein in response to detecting that the storage space available in the given block is less than the payload to be stored in the given block, allocating another block to the given stream and splitting storage of the payload between the block and the another block.

14. The method of claim 10, wherein the received packets belong to at least a first flow and a second flow, and comprising allocating different respective blocks in the receive buffer to the first flow and the second flow.

15. The method of claim 10, wherein the communication stack is included in a kernel of an operating system running in kernel space, and includes transmitting one or more payloads in the given block to the client process in user space.

16. The method of claim 10, wherein the communication stack comprises a communication program running in user space, and comprising transmitting one or more payloads in the given block directly to the client process in the user space.

17. The method of claim 10, further comprising storing a header of the received packet in a header buffer and notifying the communication stack of the stored header corresponding to the payload of the received packet in the given block.

18. The method of claim 17, further comprising generating, by the communication stack or a driver mediating between the network adapter and the communication stack, the merged packet based on the header stored in the header buffer and one or more payloads stored in the given block.

Images