PROGRAMMABLE NETWORK INTERFACE DEVICE COMPRISING A HOST COMPUTING DEVICE AND A NETWORK INTERFACE DEVICE - Patent application

Abstract

The network interface device (102) includes a programmable interface (119) configured to provide at least one bus between the network interface device (102) and the host device (101) for device interfaces. The programmable interface (119) is programmable to support multiple different types of device interfaces.

Description

[Technical Field]

[0001] Priority claim This application claims the benefit of U.S. Non-Provisional Patent Application No. 16 / 525,313, filed July 29, 2019.

[0002] Field The present application relates to a network interface device, a system comprising a host computing device and a network interface device, a method, and a computer program. [Background technology]

[0003] background Network interface devices are known and are typically used to provide an interface between a computing device and a network. Network interface devices can be configured to process data received from the network and / or process data placed on the network. Summary of the Invention [Means for solving the problem]

[0004] overview According to one aspect, a network interface device is provided, the network interface device comprising a programmable interface configured to provide a device interface over at least one bus between the network interface device and a host device, the programmable interface being programmable to support a plurality of different types of device interfaces.

[0005] The programmable interface can be configured to simultaneously support at least two instances of the device interface.

[0006] The programmable interface may comprise a common descriptor cache configured to store entries for each of transactions of a plurality of device interface instances.

[0007] Entries in the common descriptor cache may include one or more of pointer information, adapter instances and / or opaque endpoint indexes, or metadata.

[0008] The pointer information may include a pointer, a block of pointers, a linked list of pointers, or a trie.

[0009] The metadata may include one or more of an indication of whether the pointer is a pointer, points to a data location, or a pointer to a further pointer, a size associated with at least a portion of the entry, an indication of an adapter associated with the entry, an indication of one or more queues, and an indication of a position within one or more queues.

[0010] The common descriptor cache may be at least partially partitioned with different partitions associated with different device interface instances.

[0011] A common descriptor cache can be shared between different device interface instances.

[0012] A common descriptor cache can provide guaranteed cache reservations for each device interface instance.

[0013] The common descriptor cache can provide a guaranteed number of descriptors to an endpoint instance of a device interface, which may be a queue.

[0014] The common descriptor cache can be configured to prefetch one or more descriptors for one or more device interface instances.

[0015] The common descriptor cache may be configured to prefetch one or more descriptors for one or more device interfaces in response to one or more of doorbell processing for a doorbell associated with the respective device interface or the arrival of data associated with the respective device interface.

[0016] The programmable interface may comprise a scheduler configured to schedule access to said at least bus for a plurality of different device interface instances.

[0017] The programmable interface can be configured to provide each device interface instance with a respective device interface adapter.

[0018] Each device interface adapter may include a DMA engine.

[0019] The programmable interface may be configured to provide a DMA engine that is shared by the at least two device interface instances.

[0020] The programmable interface can be configured to provide a doorbell function shared by the at least two device interface instances.

[0021] The programmable interface can be configured to provide an address translation service shared by the at least two device interface instances.

[0022] At least two of the plurality of types of device interfaces may be associated with different queue structure types.

[0023] At least two of the multiple types of device interfaces may be associated with one or more different semantics or commands.

[0024] The multiple types of device interfaces may include one or more of an RDMA or Virtio or NVMe device interface.

[0025] The programmable interface may be configured to support one or more different bus protocols of the at least one bus.

[0026] One or more different bus protocols are PCIe bus, AX Iba The bus may include one or more of a CCIX bus, a CAPI bus, and an OPEN CAPI bus.

[0027] The programmable interface may comprise one or more template functions that are programmable to support one or more of the different types of device interfaces described above.

[0028] While the network interface device is operational, the programmable interface can be configured to remove at least one device interface instance.

[0029] While the network interface device is operational, the programmable interface can be configured to add at least one device interface instance.

[0030] During operation of the network interface device, the programmable interface may be configured to switch from at least one device interface instance to at least one other device interface instance.

[0031] The network interface device may be configured to receive program information configured to control programming of said network interface device.

[0032] The program information may be received from one of the host device or a network.

[0033] The program information may include one or more of firmware, RTL code, HLS code, or instruction sequences.

[0034] According to another aspect, a network interface device includes a programmable interface configured to provide a device interface over at least one bus between the network interface device and a host device, the programmable interface being programmable to support multiple different instances of the device interface.

[0035] It should be understood that any of the features of any other aspect may be used individually or in combination with the network interface device of this aspect.

[0036] According to another aspect, there is provided a data processing system comprising: a host device; and a network interface device, the network interface device comprising a programmable interface configured to provide a device interface over at least one bus between the network interface device and the host device, the programmable interface being programmable to support a plurality of different types of device interfaces.

[0037] The programmable interface can be configured to simultaneously support at least two instances of the device interface.

[0038] The programmable interface may comprise a common descriptor cache configured to store entries for each of transactions of a plurality of device interface instances.

[0039] Entries in the common descriptor cache may include one or more of pointer information, adapter instances and / or opaque endpoint indexes, or metadata.

[0040] The pointer information may include a pointer, a block of pointers, a linked list of pointers, or a trie.

[0041] The metadata may include one or more of an indication of whether the pointer is a pointer, points to a data location, or a pointer to a further pointer, a size associated with at least a portion of the entry, an indication of an adapter associated with the entry, an indication of one or more queues, and a position within one or more queues.

[0042] The common descriptor cache may be at least partially partitioned with different partitions associated with different device interface instances.

[0043] A common descriptor cache can be shared between different device interface instances.

[0044] A common descriptor cache can provide guaranteed cache reservations for each device interface instance.

[0045] A common descriptor cache provides a guaranteed number of descriptors for endpoint instances of a device interface. do An endpoint instance can be a queue.

[0046] The common descriptor cache can be configured to prefetch one or more descriptors for one or more device interfaces.

[0047] The common descriptor cache may be configured to prefetch one or more descriptors for one or more device interfaces in response to one or more of doorbell processing for a doorbell associated with the respective device interface or the arrival of data associated with the respective device interface.

[0048] The programmable interface may comprise a scheduler configured to schedule access to said at least bus for a plurality of different device interface instances.

[0049] The programmable interface can be configured to provide each device interface instance with a respective device interface adapter.

[0050] Each device interface adapter may include a DMA engine.

[0051] The programmable interface may be configured to provide a DMA engine that is shared by the at least two device interface instances.

[0052] The programmable interface can be configured to provide a doorbell function shared by the at least two device interface instances.

[0053] The host device may include at least one processor and at least one memory, said memory containing computer code of one or more programs.

[0054] The at least one memory and computer code may be configured, using the at least one processor, to cause a host computing device to write at least one doorbell to at least the doorbell functionality.

[0055] The programmable interface can be configured to provide an address translation service shared by the at least two device interface instances.

[0056] At least two of the plurality of types of device interfaces may be associated with different queue structure types.

[0057] The at least one memory and computer code can be configured to, with the at least one processor, cause the host computing device to provide one or more queues for one or more device interface instances.

[0058] At least two of the multiple types of device interfaces may be associated with one or more different semantics or commands.

[0059] The multiple types of device interfaces may include one or more of an RDMA or Virtio or NVMe device interface.

[0060] The programmable interface may be configured to support one or more different bus protocols of the at least one bus.

[0061] The one or more different bus protocols may include one or more of a PCIe bus, an AXI bus, a CCIX bus, a CAPI bus, and an OPEN CAPI bus.

[0062] The programmable interface may comprise one or more template functions that are programmable to support one or more of the different types of device interfaces described above.

[0063] While the network interface device is operational, the programmable interface can be configured to remove at least one device interface instance.

[0064] While the network interface device is operational, the programmable interface can be configured to add at least one device interface instance.

[0065] During operation of the network interface device, the programmable interface may be configured to switch from at least one device interface instance to at least one other device interface instance.

[0066] The network interface device may be configured to receive program information configured to control programming of said network interface device.

[0067] The program information may be received from one of the host device or a network.

[0068] The program information may include one or more of firmware, RTL code, HLS code, or instruction sequences.

[0069] According to another aspect, there is provided a data processing system comprising: a host device; and a network interface device comprising a programmable interface configured to provide a device interface over at least one bus between the network interface device and the host device, the programmable interface being programmable to support a plurality of different instances of the device interface.

[0070] It should be understood that any of the features of any other aspect may be used individually or in combination with the data processing system of this aspect.

[0071] According to one aspect, a method is provided that includes programming a programmable interface of a network interface device to provide a device interface over at least one bus between the network interface device and a host device, the programmable interface being programmable to support a plurality of different types of device interfaces.

[0072] The method may include programming the programmable interface to simultaneously support at least two instances of the device interface.

[0073] The programmable interface may comprise a common descriptor cache configured to store entries for each of transactions of a plurality of device interface instances.

[0074] Entries in the common descriptor cache may include one or more of pointer information, adapter instances and / or opaque endpoint indexes, or metadata.

[0075] The pointer information may include a pointer, a block of pointers, a linked list of pointers, or a trie.

[0076] The metadata may include one or more of an indication of whether the pointer is a pointer, points to a data location, or a pointer to a further pointer, a size associated with at least a portion of the entry, an indication of an adapter associated with the entry, an indication of one or more queues, and a position within one or more queues.

[0077] The method may include causing the common descriptor cache to be at least partially partitioned with different partitions associated with different device interface instances.

[0078] The method may include the step of causing the common descriptor cache to be shared between different device interface instances.

[0079] The method may include causing the common descriptor cache to provide guaranteed cache reservations to each device interface instance.

[0080] The method may include causing the common descriptor cache to provide a guaranteed number of descriptors to an endpoint instance of a device interface, the endpoint instance may be a queue.

[0081] The method may include causing the common descriptor cache to prefetch one or more descriptors for one or more device interfaces.

[0082] The method may include causing the common descriptor cache to prefetch one or more descriptors for one or more device interfaces in response to one or more of doorbell processing for a doorbell associated with the respective device interface or arrival of data associated with the respective device interface.

[0083] The programmable interface may comprise a scheduler configured to schedule access to said at least bus for a plurality of different device interface instances.

[0084] The method can include causing the programmable interface to provide each device interface instance with a respective device interface adapter.

[0085] Each device interface adapter may include a DMA engine.

[0086] The programmable interface may include a DMA engine shared by the at least two device interface instances.

[0087] The programmable interface may include a doorbell function shared by the at least two device interface instances.

[0088] The programmable interface can be configured to provide an address translation service shared by the at least two device interface instances.

[0089] The method may include causing the programmable interface to support a plurality of different queue structure types associated with respective different device interfaces.

[0090] The method may include causing the programmable interface to support a plurality of different semantics or commands associated with respective different device interfaces.

[0091] The multiple types of device interfaces may include one or more of an RDMA or Virtio or NVMe device interface.

[0092] The method may include causing the programmable interface to support a plurality of different bus protocols for the at least one bus.

[0093] The one or more different bus protocols may include one or more of a PCIe bus, an AXI bus, a CCIX bus, a CAPI bus, and an OPEN CAPI bus.

[0094] The method may include causing one or more template functions of a programmable interface to be programmed to support one or more of the different types of device interfaces.

[0095] The method may include causing at least one device interface instance to be removed while the network interface device is operational.

[0096] The method may include causing at least one device interface instance to be added while the network interface device is operational.

[0097] The method may include causing at least one device interface instance to be switched to at least one other device interface instance while the network interface device is operational.

[0098] The method may include receiving program information configured to control programming of the network interface device.

[0099] The method may include receiving program information from one of the host device or a network.

[0100] The program information may include one or more of firmware, RTL code, HLS code, or instruction sequences.

[0101] According to another aspect, there is provided a data processing system comprising: a network interface device; and a host computing device, the host computing device comprising at least one processor and at least one memory, the memory including computer code of one or more programs, the at least one memory and the computer code configured, using the at least one processor, to cause the host computing device to at least compile a program defining a device interface in association with the network interface device, the compiled program being installed on the host computing device to define a host interface for the device interface and installed on the network interface device to define a corresponding device interface.

[0102] The program may include a high-level program. The compiled program may provide firmware, HLS code, or RTL code for defining the corresponding device interface.

[0103] The compiled program can provide an mid-level program that defines at least a portion of the device driver, and the host computing device is configured to determine that at least a portion of the mid-level program can be inserted into a kernel of the host computing device.

[0104] The host interface may include a device driver. The host interface may include a dynamically loaded program.

[0105] The host interface may be at least one of verified for execution, provided as sandbox code, and provided as bytecode.

[0106] The network interface device may be as described above. The network interface device can include a programmable interface configured to provide a device interface over at least one bus between the network interface device and a host device, the programmable interface being programmable to support a plurality of different types of device interfaces. A compiled program can be installed on the programmable interface.

[0107] The programmable interface can be configured to support at least two device interface instances simultaneously.

[0108] The programmable interface may comprise a common descriptor cache configured to store entries for each of transactions of a plurality of device interface instances.

[0109] Entries in the common descriptor cache may include one or more of pointer information, adapter instances and / or opaque endpoint indexes, or metadata.

[0110] The pointer information may include a pointer, a block of pointers, a linked list of pointers, or a trie.

[0111] The metadata may include one or more of an indication of whether the pointer is a pointer, points to a data location, or a pointer to a further pointer, a size associated with at least a portion of the entry, an indication of an adapter associated with the entry, an indication of one or more queues, and a position within one or more queues.

[0112] The common descriptor cache may be at least partially partitioned with different partitions associated with different device interface instances.

[0113] A common descriptor cache can be shared between different device interface instances.

[0114] A common descriptor cache can provide guaranteed cache reservations for each device interface instance.

[0115] The common descriptor cache can provide a guaranteed number of descriptors to an endpoint instance of a device interface, which may be a queue.

[0116] The common descriptor cache can be configured to prefetch one or more descriptors for one or more device interfaces.

[0117] The common descriptor cache may be configured to prefetch one or more descriptors for one or more device interfaces in response to one or more of doorbell processing for a doorbell associated with the respective device interface or the arrival of data associated with the respective device interface.

[0118] The programmable interface may comprise a scheduler configured to schedule access to said at least bus for a plurality of different device interface instances.

[0119] The programmable interface can be configured to provide each device interface instance with a respective device interface adapter.

[0120] Each device interface adapter may include a DMA engine.

[0121] The programmable interface may be configured to provide a DMA engine that is shared by the at least two device interface instances.

[0122] The programmable interface can be configured to provide a doorbell function shared by the at least two device interface instances.

[0123] The host device may include at least one processor and at least one memory, said memory containing computer code of one or more programs.

[0124] The at least one memory and computer code may be configured, using the at least one processor, to cause a host computing device to write at least one doorbell to at least the doorbell functionality.

[0125] The programmable interface can be configured to provide an address translation service shared by the at least two device interface instances.

[0126] At least two of the plurality of types of device interfaces may be associated with different queue structure types.

[0127] The at least one memory and computer code can be configured to, with the at least one processor, cause the host computing device to provide one or more queues for one or more device interface instances.

[0128] At least two of the multiple types of device interfaces may be associated with one or more different semantics or commands.

[0129] The multiple types of device interfaces may include one or more of an RDMA or Virtio or NVMe device interface.

[0130] The programmable interface may be configured to support one or more different bus protocols of the at least one bus.

[0131] The one or more different bus protocols may include one or more of a PCIe bus, an AXI bus, a CCIX bus, a CAPI bus, and an OPEN CAPI bus.

[0132] The programmable interface may comprise one or more template functions that are programmable to support one or more of the different types of device interfaces described above.

[0133] While the network interface device is operational, the programmable interface can be configured to remove at least one device interface instance.

[0134] While the network interface device is operational, the programmable interface can be configured to add at least one device interface instance.

[0135] During operation of the network interface device, the programmable interface may be configured to switch from at least one device interface instance to at least one other device interface instance.

[0136] According to another aspect, a method is provided that includes causing a host computing device to compile a program that defines a device interface in association with a network interface device, the compiled program being installed on the host computing device to define a host interface for the device interface and installed on the network interface device to define a corresponding device interface.

[0137] The program may include a high-level program. The compiled program may provide firmware, HLS code, or RTL code for defining the corresponding device interface.

[0138] The compiled program can provide an intermediate level program that defines at least a portion of the device driver, and the method includes determining that at least a portion of the intermediate level program can be inserted into the kernel of a host computing device.

[0139] The host interface may include a device driver. The host interface may include a dynamically loaded program.

[0140] The host interface may be at least one of verified for execution, provided as sandbox code, and provided as bytecode.

[0141] The network interface device may be as described above. The method may include installing the compiled program into a programmable interface of the network interface device, the programmable interface being programmable to support a plurality of different types of device interfaces.

[0142] The method may include programming the programmable interface to simultaneously support at least two instances of the device interface.

[0143] The programmable interface may comprise a common descriptor cache configured to store entries for each of transactions of a plurality of device interface instances.

[0144] Entries in the common descriptor cache may include one or more of pointer information, adapter instances and / or opaque endpoint indexes, or metadata.

[0145] The pointer information may include a pointer, a block of pointers, a linked list of pointers, or a trie.

[0146] The metadata may include one or more of an indication of whether the pointer is a pointer, points to a data location, or a pointer to a further pointer, a size associated with at least a portion of the entry, an indication of an adapter associated with the entry, an indication of one or more queues, and a position within one or more queues.

[0147] The method may include causing the common descriptor cache to be at least partially partitioned with different partitions associated with different instances of device interfaces.

[0148] The method may include the step of causing the common descriptor cache to be shared between different device interface instances.

[0149] The method may include causing the common descriptor cache to provide guaranteed cache reservations to each device interface instance.

[0150] The method may include causing the common descriptor cache to provide a guaranteed number of descriptors to an endpoint instance of a device interface, the endpoint instance may be a queue.

[0151] The method may include causing the common descriptor cache to prefetch one or more descriptors for one or more device interfaces.

[0152] The method may include causing the common descriptor cache to prefetch one or more descriptors for one or more device interfaces in response to one or more of doorbell processing for a doorbell associated with the respective device interface or arrival of data associated with the respective device interface.

[0153] The programmable interface may comprise a scheduler configured to schedule access to said at least bus for a plurality of different device interface instances.

[0154] The method can include causing the programmable interface to provide each instance of the device interface with a respective device interface adapter.

[0155] Each device interface adapter may include a DMA engine.

[0156] The programmable interface can be configured to provide a DMA engine shared by the at least two instances of the device interface.

[0157] The programmable interface can be configured to provide a doorbell function shared by the at least two device interface instances.

[0158] The host device may include at least one processor and at least one memory, said memory containing computer code of one or more programs.

[0159] The at least one memory and computer code may be configured, using the at least one processor, to cause a host computing device to write at least one doorbell to at least the doorbell functionality.

[0160] The programmable interface can be configured to provide an address translation service shared by the at least two device interface instances.

[0161] The method may include causing the programmable interface to support a plurality of different queue structure types associated with respective different device interfaces.

[0162] The method may include having the host interface provide one or more queues for one or more device interface instances.

[0163] The method may include causing the programmable interface to support a plurality of different semantics or commands associated with respective different device interfaces.

[0164] The multiple types of device interfaces may include one or more of an RDMA or Virtio or NVMe device interface.

[0165] The method may include causing the programmable interface to support a plurality of different bus protocols for the at least one bus.

[0166] The one or more different bus protocols may include one or more of a PCIe bus, an AXI bus, a CCIX bus, a CAPI bus, and an OPEN CAPI bus.

[0167] The method may include programming, using a compiled program, one or more template functions of the programmable interface to support each device interface.

[0168] The method may include causing at least one device interface instance to be removed while the network interface device is operational.

[0169] The method may include causing at least one device interface instance to be added while the network interface device is operational.

[0170] The method may include causing at least one device interface instance to be switched to at least one other device interface instance while the network interface device is operational.

[0171] According to another aspect, there is provided a non-transitory computer-readable medium encoded with instructions that, when executed on at least one processor, enable the at least one processor to perform the steps of any of the aforementioned methods.

[0172] In another aspect, a computer program product includes computer executable code configured to, when executed, provide any of the above methods.

[0173] A computer program may also be provided comprising program code means adapted to perform the method(s). The computer program may be stored and / or otherwise embodied by a carrier medium.

[0174] A number of different embodiments have been described above, and it should be understood that further embodiments may be provided by combining any two or more of the above-described embodiments.

[0175] BRIEF DESCRIPTION OF THE DRAWINGS Some embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0176] [Figure 1] 1 is a schematic diagram of a data processing system coupled to a network; [Figure 2] FIG. 2 is a schematic diagram of a first network interface device. [Figure 3] FIG. 2 is a schematic diagram of a second network interface device. [Figure 4] FIG. 1 illustrates a first example of an adapter provided in a programmable interface of a network interface device. [Figure 5] FIG. 10 illustrates a second example of an adapter provided in a programmable interface of a network interface device. [Figure 6] FIG. 2 is a schematic diagram of another network interface device. [Figure 7] 1 is a schematic diagram of a data processing system from the perspective of memory mapping; [Figure 8] FIG. 10 illustrates a buffer table used to determine the location of a buffer. [Figure 9] FIG. 1 illustrates a method of one embodiment. [Figure 10] FIG. 10 illustrates a method of another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0177] Detailed Description The following description is presented to enable any person skilled in the art to make and use the invention, and is provided in the context of a particular application. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art.

[0178] The generic principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0179] When data is transferred between two data processing systems over a data channel such as a network, each of the data processing systems has an appropriate network interface that enables them to communicate over the channel. Often, the network is based on Ethernet technology. Other embodiments may be based on other technologies.

[0180] A data processing system that communicates over a network is equipped with a network interface that can support the physical and logical requirements of a network protocol. The physical hardware components of a network interface are referred to as a network interface device or network interface card (NIC).

[0181] A typical data processing system 100 for transmitting and receiving over a network is shown in Figure 1. The data processing system 100 includes a host computing device 101 coupled to a network interface device 102 configured to interface the host computing device to a network 103. The host computing device 101 is provided with a device driver 115 for the network interface device 102. The host computing device 101 includes an operating system 104 that supports one or more user-level applications 105. The host computing device 101 may also include a network protocol stack (not shown). For example, the protocol stack may be a component of the application, a library to which the application is linked, and / or , O In some embodiments, more than one protocol stack may be provided.

[0182] The network protocol stack may be a Transmission Control Protocol (TCP) stack. An application 105 can send and receive TCP / IP messages by opening a socket and reading and writing data to the socket. The operating system 104 can cause the messages to be transferred over the network. For example, an application can invoke a system call (syscall) to send data through the socket and then through the operating system 104 to the network 103.

[0183] Instead of implementing the stack in the host computing device 101, some systems can offload at least a portion of the protocol processing to the network interface device 102. For example, the network interface device 102 can include a TCP Offload Engine (TOE) for performing TCP protocol processing. Data transmitted over the network may be sent by the application 105 through a TOE-enabled virtual interface driver, partially or completely bypassing the TCP / IP stack in the host computing device. Thus, data transmitted along this high-speed path need only be formatted to meet the requirements of the TOE driver.

[0184] It should be understood that the TOE functionality is one example of functionality that may be offloaded to a network interface device. Alternatively, or additionally, one or more other functions may be offloaded to the network interface device. By way of example, a portion of protocol processing may be offloaded to the network interface device, an application may be provided at least in part on the network interface device, etc.

[0185] The network interface device may include one or more buffers 106. The buffers 106 may be any suitable memory, FIFO, or set of registers. The buffers may be such that any operation requiring a modification to a frame (e.g., inserting a checksum into a header) is completed before that portion of the frame is forwarded to the MAC (medium access control).

[0186] The host computing device 101 may include one or more processors 113 and one or more memories 109. (The one or more processors may actually provide at least a portion of the operating system and user-level applications along with one or more memories.)

[0187] It should be understood that the memory space of the system may be provided by one or more of: one or more memories on the host computing device, one or more memories on the network interface device, and one or more memories external to the host computing device and the network interface device.

[0188] In some embodiments, host computing device 101 and network interface device 102 may communicate via one or more buses 119. By way of example only, the bus may be a peripheral component interconnect express (PCIe) bus, an advanced eXtensible interface (AXI) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, a Coherent Accelerator Processor Interface (CAPI) bus, an OPEN CAPI bus, or any other suitable bus. If more than one bus is provided, at least one bus may use a different bus protocol than at least one other bus, or all buses may use the same bus protocol.

[0189] During operation of the data processing system, data to be transmitted over the network may be transferred from the host computing device 101 to the network interface device 102 for transmission. Data received from the data network may be passed to the host device through the network interface device.

[0190] In some embodiments, data transfers are managed using a device queue technique. The device queue technique can support DMA (Direct Memory Access). Using DMA, the network interface device can send and receive data to and from memory, bypassing the host device's CPU. Data received from the network is written directly to host memory at locations indicated by one or more descriptors (e.g., in a receive queue). With data sent over the network, the network interface transmits the data at memory locations indicated by one or more descriptors (e.g., in a transmit queue).

[0191] The host device and the network interface device communicate through a device interface. There are various different ways in which the interface between the host device and the network interface device can be implemented. The interface between the host device and the network interface device is hereinafter referred to as the device interface.

[0192] Device interfaces may be specified by industry and / or proprietary specifications, etc. Such specifications may define how a host computing device and / or a network interface device should be configured to enable the host and device to operate together.

[0193] One aspect of a device interface is how queues (sometimes referred to as rings) are defined and managed. Some device interfaces may not require a queue-type structure.

[0194] 2, which illustrates the programmable interface 510 of the network interface device in more detail. The programmable interface can be configured to support multiple different device interfaces and / or multiple different bus protocols. The programmable interface can simultaneously support one or more instances of a device interface and / or can simultaneously support one or more buses.

[0195] If two or more instances of a device interface are supported, they may support the same or different device interface specifications. The number of device interface instances may change during operation of the network interface device. The type(s) of device interfaces supported may change during operation of the network interface device.

[0196] If more than one bus is supported, they may support the same or different bus protocols. The number of buses may change during operation of the network interface device. The bus protocol of one or more buses may change during operation of the network interface device.

[0197] The programmable interface can support different device interfaces depending on the device interface requirements. For example, the device interface can use a queue or ring-pair structure TX / RX and optionally an event notification channel (or completion queue), or the device interface can use a relatively large memory window. In this latter example, frames of data are managed by incremental writes and a queue-type structure is not required.

[0198] Different device interfaces may implement the interaction between the host computing device and the network interface device in different ways.

[0199] Different device interfaces can be associated with different semantics.

[0200] In some device interface specifications, commands used to control a process may refer to a descriptor, which may identify a starting address and is sometimes referred to as a pointer.

[0201] Depending on the device interface, the host computing device can create a work queue or descriptor ring in memory. This work queue can have one or more commands. The host can transfer one or more commands from the queue to local memory on the network interface device over a communication channel. These commands can be stored in a descriptor cache (described in more detail below in the programmable interface section).

[0202] A network interface device can send and receive data from a network. An exemplary device interface specification can set up two work queues in memory: a transmit (TX) work queue that identifies data buffers in memory that are ready to be sent onto the network, and a receive (Rx) work queue that identifies data buffers in memory that are available to receive incoming data from the network.

[0203] As previously mentioned, there may be various ways of managing descriptors depending on the device interface. In some embodiments, the programmable interface of a network interface device is configured such that the same network interface device can support any one or more of multiple different device interface specifications. For example, multiple different ways of implementing descriptors may be supported by the programmable interface.

[0204] Different device interface specifications and / or queue management and / or queue structure semantics may differ. For example, different specifications may have different commands and / or different semantics for certain commands and / or different actions for different commands. In some specifications, a separate completion queue may be provided, while in other specifications, a completion queue is not provided. Different device interfaces may require different metadata. Different device interfaces may use different checksum mechanisms (e.g., using different calculation techniques and / or calculating over different data sets). Different device interfaces may have different ways of providing verified checksum information. By way of example only, the information may be provided as a prefix to the packet data or as part of the completion.

[0205] Specifications may differ in how frames of data are handled with respect to buffers. For example, a buffer may receive only one frame of data, a buffer may receive multiple frames of data, or a single frame may span two or more buffers. This may be defined by the particular implementation of the device interface specification or the host specification.

[0206] The device interface may support indirect descriptors, where one descriptor in a ring points to a table of more descriptors rather than to a data buffer. This may be used, for example, to "stretch" the ring. This type of configuration may be supported, for example, by Virtio, described below.

[0207] One or more device interface specifications can define how and / or when transactions are placed on the bus. These may be, for example, DMA-related transactions. For example, some specifications may aggregate transactions in some or all circumstances. Some specifications may specify that single transactions are placed on the bus separately. Some specifications may define attributes of transactions placed on the bus, such as alignment and / or padding.

[0208] A device interface may be a "standardized" interface, a proprietary interface, or even an implementation-specific interface. By way of example only, some approaches for implementing a device interface are provided by RDMA (remote direct memory access), Virtio, and NVMe (Non-Volatile Memory Host Controller Interface Express). NVMe is an open logical device interface specification for accessing non-volatile storage media attached via a PCI Express (PCIe) bus.

[0209] Virtio devices are visible to the virtual environment. By design, they appear as physical devices to guests within a virtual machine. The Virtio specification treats them like physical devices. This similarity allows guests to use standard drivers and discovery mechanisms. The goal of the Virtio specification is that virtual environments and guests should have a standard and extensible mechanism for virtual devices. Virtio devices may be implemented in hardware as physical interfaces. This allows guest operating system images to run in a hypervisor-less environment.

[0210] The EF 100 provided by the present applicant is an example of a so-called proprietary specification. It should be understood that these examples of different device interfaces are merely examples and that other embodiments may alternatively or additionally be used with any suitable device interface.

[0211] The specification may define virtualization behavior and / or hardware behavior and / or software behavior and / or firmware behavior.

[0212] The programmable interface is configured so that the requirements of at least two different device interfaces can be supported by the same programmable interface.

[0213] As mentioned above, different device interfaces may have different requirements for implementing, for example, queue-type structures. For example, the layout and functionality of at least some queues may be broadly similar, but there may be some differences. For example, one or more of the descriptor format and layout, pointer indexes, command and notification completions may differ.

[0214] For example, the EF 100 has a receive queue, a transmit queue, and an event queue. Some device interface specifications may have a set of network interface device-host rings, host-network device rings, and completion rings. As an example, a transaction queue may be composed of the following elements: descriptor rings, completion rings, data buffers, doorbells, and interrupt targets.

[0215] An example device interface specification may define several user-configurable host profiles, each specifying which of two memory regions holds a particular queue element. In some embodiments, the number of user-configurable host profiles is 16. It should be understood that this is merely an example and other implementations may have a different number of user-configurable host profiles.

[0216] For example, a PCI physical or virtual function may be specified to support several Virtio queues (VRING) and have a Virtio-net personality. Another function may support NVMe queue pairs and have an NVM e It can have a personality.

[0217] Every queue can be assigned one of the pre-configured host profiles.

[0218] A profile can specify that queue data, rings, and interrupts reside in a first host memory space. Similarly, a queue can reside in a second host memory space. A queue can have its elements in different spaces. For example, data may come from a first host memory, but descriptors, completions, and interrupts reside in a second host memory, or vice versa.

[0219] The device interface adapter 308 (described in more detail below) can enable the host-network interface queue descriptor to select a data location that overrides the data location specified in the host profile for the queue.

[0220] A device interface adapter can support the DMA Host Profile feature by storing a queue's assigned profile ID in a local queue state table and supplying the ID as part of adapter DMA transactions as needed.

[0221] RDMA has a set of queue pairs. In some embodiments, this set may include 32k-128k queue pairs. Each queue pair can support a send queue (SQ), a receive queue (RQ), and a completion queue (CQ).

[0222] Virtio can have one or more pairs of VIRT queues and optionally control VIRT queues. In some embodiments, a Virtio block device provides a block storage interface through a single VIRT queue. The driver sends read requests that transfer blocks of data from storage to the driver and write requests that transfer blocks of data from the driver to the device. A Virtio device may require a descriptor table, available rings, and used rings.

[0223] An instance of a device interface provides a communication channel between a host software entity and a hardware data path.

[0224] The programmable interface 510 may be provided at least in part by configurable hardware. The configurable hardware may be any suitable configurable hardware. By way of example only, the programmable interface 510 may be provided by one or more of an FPGA and an ASIC microengine. The operations to be performed by the programmable interface may be specified in a high-level language, which may then be compiled to gate level or RTL (register transfer language). Alternatively, the operations may be provided as firmware programmed into a microprocessor such as an ASIC. Programs such as HLS (high-level synthesis) may also be used.

[0225] The same programmable interface 510 may, in some embodiments, be programmed to support any one or more suitable bus protocols and / or any one or more suitable device interface specifications, whether when the network interface device is first used and / or while the device is in use.

[0226] The programmable interface 510 has an interface 304 with one or more shared buses 119. (By way of example only, FIG. 2 shows two buses.) The buses 119 are provided between the host device and the network interface device. Depending on how the network interface device is deployed, the interface 304 can be configured to interface with one or more buses 119. When there are two or more buses, the buses can operate according to the same or different protocols, as described above. In some embodiments, the interface may be configured to support one or more buses that can alternatively be divided into smaller units (e.g., bifurcated or quad-furcated).

[0227] 2 also shows the streaming subsystem 310 having one or more data slices 512. A slice can perform data processing on the data, for example, before the data is transmitted or after the data is received.

[0228] The programmable interface 510 is configured to implement one or more adapters 308. An adapter can support an instance of a device interface. In some embodiments, an adapter can support two or more instances of a device interface. When two or more adapters are implemented, different adapters can be configured to support different device interfaces. In some embodiments, two or more adapters may use the same device interface specification.

[0229] The adapter is connected to the host computer. Ting (compute) Device and / or host computer Ting It can be configured to execute and control the respective operations from the device, which may be after hardware virtualization.

[0230] The adapter may be for one or more of communication to and from the host device. The adapter can support one or more of DMA transactions to and from the host (e.g., for data received from the network, data sent over the network, and / or information transferred via DMA as part of the virtual device interface, such as descriptor fetches).

[0231] One or more adapters can receive input from and / or provide output to one or more slices 512. A slice may be part of the streaming subsystem 310. In some embodiments, a given slice can communicate with a single adapter or two or more adapters.

[0232] The programmable interface 510 has a common scheduler function 306 that controls access to one or more buses. The scheduler function may, for example, control access to a shared bus between a network interface device and a host. A bus has a finite bandwidth that needs to be shared, for example, between different instances of a device interface. For example, the scheduler may be configured to ensure that each transaction of a different instance of a device interface is supported. The scheduler must schedule bus resources (e.g., DMA bandwidth) among all transactions. Similar to device interface instances, the programmable interface may support other DMA clients (e.g., user-defined external kernels). The scheduler may also be configured to schedule these functions.

[0233] In some embodiments, a single scheduler function is configured to manage the scheduling.

[0234] The scheduler can also schedule device resources, such as buffers in data path slices. The scheduler can be used to enforce quality of service policies between virtual device interfaces. The quality of service policies can relate to one or more of per-queue priority or bandwidth, for example. The scheduler can manage fairness between different subsystems of the network interface device.

[0235] Adapter transactions are scheduled by scheduler 306, which in some embodiments may also manage any other necessary access.

[0236] The programmable interface comprises a doorbell function 322. All doorbells associated with different instances of the device interface from the host to the network interface device are written to this common doorbell function. The use of doorbells and / or how they are implemented and / or how they are used may depend on the respective device interface. For example, the host may provide a doorbell when the host has data to be written. The host may provide a doorbell to indicate a receive queue. More generally, the host may provide a doorbell to the doorbell function when it desires to do something via the programmable interface 510.

[0237] The doorbell function can be configured to determine the adapter for which the doorbell is intended by the base address of the doorbell write.

[0238] The doorbell itself can be a relatively large data frame, or can be relatively short, indicating, for example, a particular queue or an index within a queue. For example, a relatively large doorbell can contain data transmitted in either the first part of a frame or the entire frame. When a doorbell is written to the doorbell function, this causes adapters 308 that support the instance of the device interface for which the doorbell is intended to respond to the doorbell appropriately.

[0239] A doorbell may typically be generated by a host CPU executing an instruction such as a load / store to a destination address that is mapped so that a target bus write is issued. Bus mapping attributes (e.g., write combining) and the specific instruction used (e.g., write memory barrier) affect the bus transaction that is generated.

[0240] The doorbell function can enforce doorbell behavior, which may include flagging abnormal doorbells and / or rejecting faulty doorbells. The doorbell function can, for example, merge two or more doorbells associated with a given adapter.

[0241] The programmable interface comprises an address translation service ATS 320, which is described in more detail below. The ATS can be shared by different instances of the device interface. The programmable interface is configured to allow an IOMMU and / or an ATS to be used. This may depend on the device interface specification and / or the particular type of DMA access. The programmable interface can be configured to support either or both an IOMMU and an ATS.

[0242] In some embodiments, a common DMA descriptor cache 314 may be provided by the programmable interface. This descriptor cache may contain, for example, all cached descriptors for all device interface instances. The descriptor cache may have pointers that resolve to bus physical addresses. It should be understood that if the cache is beginning to run out of descriptors for a respective queue, the cache may fetch one or more descriptors for each queue. This may be controlled by the descriptor cache and / or the respective adapter.

[0243] The descriptor cache can evict one or more descriptors, for example, if the descriptor is stale or to make room for other descriptors. The eviction may be based on one or more of random, least recently used, a priority associated with the adapter, the number of descriptors associated with each adapter, and / or any other suitable criteria.

[0244] The operation of the descriptor cache may take into account one or more of service guarantees and usage patterns in determining which descriptors from which queues to fetch and / or which descriptors to remove from the descriptor cache when or if the descriptor cache is full. The descriptor cache may attempt to pre-fetch descriptors in advance to avoid head-of-line blocking.

[0245] The descriptor cache may be partitioned among different adapters. The partitions may be such that each adapter has a fixed allocation. The allocation for each adapter may be the same or different. Alternatively, the size of the partitions may vary, which may depend on factors such as the activity of one or more adapters and / or the type of activity and / or device interface type supported by the adapters.

[0246] Alternatively, the descriptor cache may be shared between different adapters depending on the adapters' requirements.

[0247] In some embodiments, the descriptor cache may have a fixed quota associated with each adapter and a shared quota that may be used by any one or more of the adapters as needed.

[0248] The descriptor cache can provide guaranteed cache reservations for each adapter instance.

[0249] A descriptor cache can provide a guaranteed number of descriptors to an adapter's endpoint instance, which may be a queue.

[0250] The descriptor cache may be configured to prefetch one or more descriptors in response to one or more of doorbell processing of a doorbell associated with the respective adapter and / or the arrival of data associated with the respective adapter.

[0251] Reference is now made to FIG. 3, which shows modifications to the programmable interface 510 of FIG. 2 in more detail.

[0252] In this example, the programmable interface is provided with multiple interfaces. In this example, two are shown. However, in some embodiments, more than two interfaces may be provided. It should be understood that the configuration of Figure 2 may be modified to have two interfaces.

[0253] 3, a shared DMA engine 330 is provided. This DMA engine is shared by different adapters 308. A shared DMA engine may also be used with a single interface, as shown in FIG.

[0254] As will be described below, whereas in the configuration of Figure 2 each adapter is provided with a DMA engine, in the example of Figure 3 the DMA engine is shared by two or more adapters. A shared DMA engine can be advantageous in that resources are used efficiently. The DMA engine provided to a particular adapter can be optimized for the behavior of that adapter.

[0255] The DMA engine reads DMA commands from the descriptor cache and executes them in a defined sequence. The DMA engine reads data from or writes data to memory. The DMA engine interfaces with the bus interface and can handle bus flow control protocol mechanisms such as credits, out-of-order completion, and parallel transactions (e.g., multiple outstanding transactions). The allocation of bandwidth available to the DMA engine is controlled by a scheduler.

[0256] Reference is now made to FIG. 4, which illustrates an example of the adapter 308 of FIG. 2 in more detail. The adapter 308 includes a doorbell handler 350 , a descriptor loader 354 , a DMA engine 356 , and RX / TX processing functionality 360 .

[0257] When a doorbell is received by the doorbell function 322, it is resolved based on a dynamic address map to a particular adapter. A doorbell handler 350 for a particular adapter parses the doorbell. The doorbell handler can be configured to poll the doorbell function to obtain doorbells for that particular adapter. The parsing determines one or more of a command, a descriptor, data, and a device index associated with the doorbell. The command may be any suitable command, such as a data read, data write, and / or a more complex command.

[0258] The doorbell handler provides output including one or more portions of the parsed doorbell to a descriptor loader. The descriptor loader is configured to assemble a descriptor from a descriptor cache. The descriptor loader is also configured to fetch cache context information, such as a queue pointer. The cache context information, such as a queue pointer, can provide the state of a descriptor index.

[0259] The descriptor cache provides requested descriptor cache lines to the descriptor loader. The descriptor cache can directly access (DMA) descriptor cache lines from one or more of the host or device memory. The descriptor cache can prefetch indirect descriptors and one or more of other descriptors. The descriptor cache can evict outdated descriptors.

[0260] The command and associated descriptor(s) are provided to the descriptor handler by the descriptor loader. The descriptor handler converts the command(s) into a DMA request chain. Thus, the chain of operations culminates in a list of DMA requests to the DMA engines. The DMA engines execute the DMA requests. The DMA requests are scheduled by the scheduler 306.

[0261] The DMA engine is programmed with the semantics of a particular device interface. Several DMA responses may be required for the processing of a descriptor. The descriptor handler is aware of the DMA engine's progress through one or more completions provided to it by the DMA engine.

[0262] Other DMA responses may be delivered to the RX / TX processing function, which is configured to perform functions such as offload processing, reassembly logic, etc.

[0263] It should be understood that Figure 4 shows one adapter. It should be understood that the programmable interface can support one or more adapters.

[0264] Refer to Figure 5, which shows an example of adapter 308 of Figure 3 in more detail. The adapter of Figure 5 operates in much the same way as the adapter of Figure 4. However, in this case the DMA engine is external to the adapter and can be shared by two or more different adapters. However, the descriptor handle Dora and interaction with the DMA engine by the Rx / TX functions is the same as that described in connection with Figure 4. In these embodiments, the scheduler is configured to schedule DMA accesses by different adapters 308 through a common DMA engine.

[0265] Referring to Figure 7, which illustrates aspects of memory address management in some embodiments, the host computing device 101 is provided with a device driver 115 for the network interface device 106. The device driver 115 effectively makes a portion of memory 109 within the host accessible to the network interface device for DMA reads and writes.

[0266] The device driver 115 has a "top edge" referenced 590 that conforms to the operating system's device interface API. The operating system device interface API can be NDIS or the Linux® Network Driver API. The device driver 115 has a "bottom edge" referenced 591 that conforms to the network interface device hardware and performs register-level operations to perform operations such as API operations, send operations, receive operations, interrupt operations, and buffer ring refill operations.

[0267] The lower edge can support vendor or device-specific operations. In some embodiments, a virtual device interface is standardized to make device drivers generic device types (e.g., network device drivers). Virtio and NVME are such examples.

[0268] The host may include an IOMMU (input / output memory management unit) 514. The IOMMU may be configured to provide memory mapping on the host side. In some embodiments, memory mapping occurs on the host side of the bus 119. Whether such an IOMMU is provided and / or how it operates may be part of the host CPU and / or chipset architecture.

[0269] Address translation may be performed by an IOMMU. A network interface device can "write" data to an address in a descriptor queue. The write transaction is received by the IOMMU and then written to the physical location associated with the address in the descriptor queue. The IOMMU technique may be used for some DMA accesses by the network interface.

[0270] The programmable interface may support an address translation service (ATS) or buffer table. The ATS translates addresses that the device can use for DMA transactions and may form part of a descriptor queue. Addresses may be translated to bus or physical addresses. If an address translates to a bus physical address, the IOMMU may translate the bus physical address. The ATS may act as a cache for the IOMMU, or may map buffer virtual addresses to bus physical addresses, or may map buffer virtual addresses to physical addresses where no IOMMU exists. This may be defined at least in part by the required device interface behavior.

[0271] For one or more device interface types, ATS may not be provided. It should be understood that how address translation is managed may be defined by the device interface, and thus the mapping of addresses in the descriptor to physical locations may be performed in the IOMMU and / or the ATS depending on the device interface.

[0272] In some embodiments, an address space for each application can be provided. A descriptor queue associated with a particular application can have a specific address. In some embodiments, the ATS can be configured to translate that address. Note that a given queue cannot generally reference memory space associated with another queue. The host's OS can update the buffer table. An application thread can post a receive descriptor, which defines the memory into which the application wants to inject its data, for example, or the memory from which data to be transmitted is fetched. The network interface device determines the physical memory location associated with the indicated memory location using, for example, a mapping using the ATS or the buffer table.

[0273] DMA allows data to be transferred to and / or from regions of the address space allocated to the network interface device, such that data received via the receive slice 512 of the network interface device can be written to host memory via a DMA process, and data in host memory can be transmitted via the transmit slice 512 of the network interface device, also via a DMA process.

[0274] The mapping may be implemented as a private address mapping for the context of the descriptor queue, or as a shared data structure such as a system page table, or as an address translation cache.

[0275] When the network interface device injects data into a physical location, it can update the event queue of the associated socket and the write pointer in the receive queue used by the network interface device. The network interface device can write or inject the next received data into the next location indicated by the updated write pointer in the receive queue. The pointer is stored in a descriptor cache.

[0276] An example of a buffer table is shown in Figure 8 and is used to determine the physical location (or bus physical address as described above) of a buffer. An IOMMU or ATS may be implemented as a buffer table. In some embodiments, the buffer table maps which thread or application is assigned to each buffer and holds the following information for each buffer:

[0277] 1. Buffer index. This is the buffer's identification number, which is unique to the host. The buffer is identified by this number in the buffer table index field.

[0278] 2. Ownership, which indicates which application or thread owns the buffer.

[0279] 3. Physical memory address: The buffer occupies a contiguous block of physical memory, and this number indicates where that block begins (or the bus physical address as mentioned above).

[0280] 4. Length: This indicates the length of the buffer in memory. The size of each buffer table entry may be selected to match the host system's page size, typically 4 kilobytes. In this way, contiguous regions of an application's virtual address space that are mapped to discontiguous regions of physical memory can be represented by a sequence of buffer table entries. In other embodiments, sizes up to gigabyte Huge Page sizes may be supported.

[0281] The mapping of the indicated memory to physical memory locations may be performed using a buffer table.

[0282] The descriptor cache can contain direct or indirect indications of which adapters a descriptor is associated with, allowing the correct semantics to be applied to each descriptor.

[0283] In some embodiments, a given entry in the descriptor cache: A pointer to the location where data should be written / read, the associated adapter instance and / or opaque endpoint index, and Metadata may include one or more of:

[0284] The pointer may be a single pointer, a block of pointers, a linked list of pointers, a trie, or any other suitable data structure.

[0285] The metadata can include one or more flags. The metadata can indicate whether the pointer is to a table of descriptors rather than a buffer, and / or whether the pointer is to a buffer.

[0286] The metadata may indicate the device interface type and / or size. In some embodiments, the descriptors may vary in size. In some embodiments, there may be a set of different sizes. The metadata may indicate which size is being used.

[0287] In some embodiments, the metadata may indicate a device interface type.

[0288] The metadata may indicate one or more queue indices (the position of the descriptor relative to the logical queue) and / or associated queues.

[0289] It should be understood that the metadata, if provided, depends on the requirements of the associated device interface. The format of the metadata may depend on the requirements of the associated device interface. The metadata may be treated as opaque by the descriptor cache.

[0290] The device index can provide information about the application the descriptor is associated with. This may indicate an adapter or may be provided separately for a given entry. The device index can provide information about whether the entry is for a receive queue, a command queue, a transmit queue, or even a window.

[0291] To track the queue and / or window states of many user-level applications that may be communicating with the network interface device simultaneously (and possibly using different device interface specifications), a common descriptor cache may include at least the following descriptors: The descriptor cache may be configured to hold at least several descriptors for each endpoint. This allows, for example, an adapter to ensure that frame delivery is never stalled due to descriptor fetching because at least several descriptors are always available. By way of example, the number of descriptors may be 0, 1, 16, 32, 64, or 128 descriptors. Different endpoints may be associated with the same or different numbers of descriptors.

[0292] Each device interface may have at least one corresponding device index that may be used as an index into a common descriptor cache.

[0293] The host may be configured to download one or more device drivers to support one or more of the device interface types supported by the programmable interface of the network interface device.

[0294] In some embodiments, the network interface device may provide device drivers for one or more of the different interface types supported by the network interface device. The device drivers received by the host may be expressed as device drivers or as programs, such as eBPF programs.

[0295] In some embodiments, the device driver and device interface are generated from a single high-level program that is compiled and downloaded to the network interface device instance by the operating system. The host computing device can compile the program to provide firmware or RTL level code for the device interface and kernel modules, device drivers, or any other representation suitable for the host interface of the native operating system.

[0296] The host computing device can be configured to compile a high-level language program into an intermediate-level language program for a host interface (device driver). This intermediate-level language program can be verified if the program is to be inserted into a higher-privilege domain, such as the kernel. By way of example only, the intermediate-level language program may be an eBPF (or extended eBPF) program.

[0297] The host computing device can be configured to compile high-level language programs into intermediate-level language programs for device interfaces (e.g., HLS, or firmware for running a physical device). The source of the high-level program may be an application with a higher level of privilege or a lower level of privilege. An application with a higher level of privilege may be a so-called trusted application, and therefore is entrusted with making kernel-level and other modifications. An application with a lower level of privilege may be a so-called untrusted application, and therefore is not able to make kernel-level and other modifications. However, the compiled output of an untrusted application may be verified as safe for use by a privilege verifier or otherwise executed in a sandbox environment.

[0298] Applications can express the semantics of a device interface using a high-level language that is compiled into a device driver in the host and a device interface in the network interface device. Depending on the privileges associated with the application, the device driver can be inserted into the kernel (with higher privileges) or outside the kernel (with lower privileges).

[0299] Programs for network interface devices used by the programming interface to provide the required device interface may be associated with higher or lower privileges. If a program has higher privileges, it can use any or all of the hardware features associated with the higher privileges. If a program has lower privileges, it can only use hardware associated with the lower privileges. As examples, if a program is associated with a lower level of privilege, access may be limited to certain DMA buffers, and / or the number of interrupts allowed may be limited, and / or no interrupts may be allowed, and / or scheduling of access to the bus may be limited.

[0300] In some embodiments, a device interface may be provided within a sandbox environment on a network interface device. For example, an adapter may be provided within the sandbox environment. The sandbox environment may be used with an adapter associated with a lower level of privilege.

[0301] In some embodiments, the programming of the ATS may be controlled according to the privilege associated with a given adapter to ensure that a lower privilege adapter is not given access to buffers associated with higher levels of privilege, or is given only buffers that the adapter is allowed to access.

[0302] In some embodiments, the programmable interface may be provided with template functions, such as one or more of the scheduler function, doorbell function, ATS function, descriptor cache, DMA engine (if shared), and multiple adapters, as templates. When program code or the like is received by the programmable interface for a particular device interface, it causes the adapter's template to be modified to match the particular device interface. One or more of the scheduler function, doorbell function, ATS function, descriptor cache, and DMA engine templates may be modified to support the particular device interface.

[0303] In some embodiments, the programming of the programmable interface may be performed at run time, hi some embodiments, the programming of the programmable interface may be performed during operation of the host computing device and the network interface device.

[0304] The foregoing embodiments have described a programmable interface for use between a network interface device and a host device. It should be understood that, alternatively or additionally, the programmable interface may be used for routing on the network interface device. For example, there may be components on the network interface device that are configured to communicate using a bus. That bus may be any of the types of buses described above.

[0305] For example, a network interface device may include two or more ASICs configured to communicate using a bus, such as a PCIe bus. In some embodiments, the same programmable interface configured to interface with a host device may also be configured to interface between two or more components on the network interface device. In other embodiments, a separate programmable interface may be used to interface with the host device, as compared to that required for interfacing between components on the network interface device.

[0306] By way of example only, some examples of exemplary network interface devices including a programmable interface as described above will now be described with reference to FIG.

[0307] In some embodiments, protocol processing may be offloaded to a network interface device to be performed within the network interface. If such offloading is performed in the network interface device, it may be desirable to implement functionality on the network interface device that can utilize the protocol processing capabilities of the network interface device. By implementing such functionality in the network interface device instead of the host, potential improvements in latency can be achieved by reducing data transfers between the host and the network interface device.

[0308] The network interface device 102 includes a transport engine 215 configured to process data packets according to a transport protocol, such as TCP / IP. The transport engine 215 may include a protocol stack. The transport engine 215 may include multiple slices 512 or data pipelines, where a portion of the slices are RX slices configured to perform receive processing for ingress data packets received from the network 103 and a portion of the slices are TX slices configured to perform transmit processing for egress data packets to be transmitted on the network 103. In some embodiments, the slices may be capable of processing both transmitted and received data.

[0309] In the illustrated example, four slices are provided. However, it should be understood that in other embodiments, a different number of slices is used. In one embodiment, a slice can be configured to process receive data or to process transmit data. In other embodiments, a slice may be configured to be able to process receive data and transmit data. In some embodiments, the number of slices may be the same as the number of ports. In some embodiments, there may be a transmit slice and a receive slice for each port. In some embodiments, there may not be a direct correlation between the number of ports and the number of slices. In some embodiments, a slice can dynamically switch from processing receive data to processing transmit data and vice versa.

[0310] Each slice can be thought of as a processing engine. As such, it can execute microcode to perform functions such as parsing data paths, matching offloads, and delivering semantics. A slice can operate on any bit of a frame.

[0311] A slice can perform analytical operations on the data it is processing. There may be matching actions that match data with, for example, filters and action functions that perform actions or that do not depend on the results of the match.

[0312] Network interface devices can provide features such as flow steering and low latency operation, hardware time stamping and clock synchronization.

[0313] Ethernet interface 225 is configured to receive data packets from the network and pass them to Layer 2 streaming interface 220. Layer 2 streaming interface 220 is configured to pass the data packets to transport engine 215, which performs processing on the data packets, before passing the processed data packets to programmable interface 510. Programmable interface 510 is configured to pass at least a portion of the data to host 101, for example, in a DMA write operation. The reverse of this path can be followed for data to be sent over the network. The protocol-processed data packets are then passed to the transport engine 215 via the programmable interface 510. Q1 Layer 2 before being sent to Layer 3 Streaming The data may be passed to interface 220 and then to Ethernet interface 225.

[0314] 9, which illustrates a method of some embodiments, in which: In step S1, a programmable interface of a network interface device is programmed to provide at least one bus between the network interface device and a host device for a device interface, and the programmable interface is programmed to support a plurality of different types of device interfaces.

[0315] Referring to Figure 10, which illustrates a method of some embodiments, in step T1, a host computing device compiles a program that defines a device interface in relation to a network interface.

[0316] In step T2, the compiled program is installed in a host computing device to define a host interface for the device interface, and is installed in a network interface device to define a corresponding device interface.

[0317] As used herein, "identifying" an item of information does not necessarily require direct specification of that item of information. For example, a pointer that "identifies" a starting address in memory may specify an entire physical memory address, or may specify an address in a larger memory address space that is mapped to the physical address, or may specify a virtual address that is mapped to the physical address, or may specify an address in a user address space that is mapped to the physical address further depending on some kind of user ID, or may identify an address that is one less than or greater than the actual starting address identified in any of these ways, etc.

[0318] The host computing or processing device or system may be any suitable computing device or system. By way of example, the host computing device may include one or more of a server, a personal computer, a switch, a router, a bridge, a virtualization system (e.g., a virtualization operating system and its supporting applications), a portable device such as a tablet computer or smartphone, or any other suitable computing device. In some embodiments, the host computing device may be provided by a set of two or more computing devices. These computing devices may be the same or different. These computing devices may be any one or more of the aforementioned examples of computing devices.

[0319] A network interface device may be any hardware device configured to interface a wired or wireless network to a host computing device. The network interface device may be provided in any manner between the network and the host device. The network interface device may be an integral part of the host processing device. The physical hardware components of the network interface need not be in the form of a card, but are sometimes referred to as a network interface card (NIC). For example, they may be in the form of an integrated circuit (IC) and connector mounted directly on a motherboard. The network interface device may be provided by a LAN (local area network) or LOM (LAN on motherboard). Alternatively or additionally, the network interface device may be provided as one or more peripheral devices, such as a PCIe card, of the host computing device. It should be understood that in other embodiments, any other suitable network interface device may be used.

[0320] The network interface device may be implemented by at least one processor and at least one memory.

[0321] The embodiments may be implemented by computer software executable by a data processor, such as one in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs, also referred to as program products, including software routines, applets, and / or macros, may be stored on any device-readable data storage medium and include program instructions for performing specific tasks. A computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one software code or portions thereof.

[0322] Further, in this regard, it should be noted that any block of the logic flow as shown may represent a program step, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. Software may be stored on physical media such as memory chips or memory blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. Physical media are non-transitory media.

[0323] The memory may be of any type suitable for the local technology environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed and removable memory, etc. The data processor may be of any type suitable for the local technology environment and may include, by way of non-limiting examples, one or more of a general purpose computer, a special purpose computer, a microprocessor, and a digital signal processor (DSP).

[0324] The applicant hereby discloses each individual feature described herein and any combination of two or more such features, whether or not they solve the problems disclosed herein, alone, and to the extent that such feature or combination can be implemented based on the specification as a whole in light of the common general knowledge of those skilled in the art, without being limited to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the above description, it will be apparent to those skilled in the art that various modifications can be made within the scope of the present invention.

Claims

1. 1. A network interface device, comprising: a programmable interface configured to provide at least one instance of a device interface over at least one bus between the network interface device and a host device, the programmable interface being programmable to support a plurality of different types of device interfaces; While the network interface device is in operation, the programmable interface Deleting at least one instance of the device interface; Adding at least one instance of a device interface; or Switching from at least one instance of a device interface to at least one other instance of a device interface A network interface device configured to perform at least one of the following:

2. The network interface device of claim 1 , wherein the programmable interface is configured to simultaneously support at least two instances of a device interface.

3. 3. The network interface device of claim 2, wherein the programmable interface comprises a common descriptor cache, the common descriptor cache configured to store entries for each of transactions of at least two instances of the device interface.

4. The entries in the common descriptor cache include Pointer information, Adapter instance and / or opaque endpoint index, or Metadata The network interface device of claim 3 , comprising one or more of:

5. The metadata includes: an indication of whether the pointer information is a pointer, a pointer to a data location, or a pointer to a further pointer; a size associated with at least a portion of said entries; an indication of an adapter associated with said entry; An indication of one or more cues; and Position in one or more queues The network interface device of claim 4 , comprising one or more of:

6. The common descriptor cache comprises: partitioned at least in part by different partitions associated with different device interface instances; be shared between different device interface instances; providing a guaranteed cache reservation for each device interface instance; and Providing a guaranteed number of descriptors for an endpoint instance of a device interface The network interface device according to any one of claims 3 to 5, configured to perform one or more of the following:

7. 7. The network interface device of claim 3, wherein the common descriptor cache is configured to prefetch one or more descriptors for one or more device interfaces in response to one or more of a doorbell process for a doorbell associated with the respective device interface or an arrival of data associated with the respective device interface.

8. 8. A network interface device according to claim 2, wherein the programmable interface comprises a scheduler configured to schedule access to the at least one bus for at least two instances of the device interface.

9. A network interface device according to any one of claims 2 to 8, wherein the programmable interface is configured to provide each device interface instance with a respective device interface adaptor.

10. 10. The network interface device of claim 9, wherein each device interface adapter comprises a DMA engine.

11. The programmable interface includes: a DMA engine shared by at least two instances of said device interface; A doorbell function shared by at least two instances of the device interface; and An address translation service shared by at least two instances of said device interface. A network interface device according to any one of claims 2 to 10, configured to provide at least one of:

12. 12. The network interface device of claim 1, wherein at least two of the multiple types of device interfaces are associated with one or more of different queue structure types and one or more of different semantics or commands.

13. The network interface device of any preceding claim, wherein the programmable interface is configured to support one or more different bus protocols of the at least one bus.

14. A network interface device according to any preceding claim, wherein the programmable interface comprises one or more template functions programmable to support one or more of the different types of device interfaces.

Images