Cross bus memory mapping

Abstract

The present disclosure relates to cross bus memory mapping. A computerized system for efficient interaction between a host having a first operating system and a second operating system, the system including a subsystem on the second operating system, the subsystem directly fetching data from a buffer local to the host, wherein the system is operable to map memory from one bus associated with the first operating system to a different bus associated with the second operating system, and to access the memory from the different bus, thereby emulating a connection between the first operating system and the second operating system through cross bus memory mapping.

Description

Technical Field

[0001] This invention relates generally to hardware storage devices, and more specifically to networked hardware storage devices. Background Technology

[0002] For example, zero-copy within a single operating system is known through the use of the Linux TCP stack.

[0003] PCIe (Peripheral Component Interconnect High Speed) (also abbreviated as PCI-e) is an example of a high-speed serial computer expansion bus standard, as described in particular in the Wikipedia entry on PCI_Express.

[0004] The following article, lwn.net / Articles / 752188, available online, describes the MSG_ZEROCOPY feature added in Linux kernel 4.14 that enables zero-copy transmission of data. It adds that "the implementation of zero-copy reception will...have to map...packet buffers into user-space memory."

[0005] and It is known product.

[0006] These are next-generation cloud SmartNIC and I / O processing unit (IPU) solutions, delivering unprecedented data center security, performance, and efficiency at scale for any workload. Continuing Mellanox's innovation in high-performance cloud architectures, Dx and By introducing a tangential hardware acceleration engine and best-in-class software programmability, they make the impossible possible for mission-critical applications. They offer new levels of security and efficiency with the highest return on investment (ROI) for hyperscale, enterprise, and telecom environments.

[0007] The Dx SmartNIC offers up to two ports of 25, 50, or 100 Gb / s or a single port of 200 Gb / s, Ethernet connectivity powered by 50 Gb / s PAM4 SerDes technology, and PCIe 4.0 host connectivity. The Dx hardware offloading engine includes IPsec and inline TLS dynamic data cryptography, advanced network virtualization, RDMA over converged Ethernet (RoCE), and structural NVMe (NVMe-oF) storage acceleration.

[0008] The IPU will be integrated into a single System-on-Chip (SoC). All of DX's advanced capabilities are integrated with a powerful ARM processor core, a high-performance memory interface, and an array of flexible processing power.

[0009] RDMA for data centers based on Ethernet or hybrid protocol architectures is described online here: Mellanox.com / sites / default / files / related-docs / whitepapers / roce_in_the_data_center.pdf.

[0010] Wikipedia states: "In computing, Remote Direct Memory Access (RDMA) is direct memory access from one computer's memory to another computer's memory without involving either computer's operating system. This allows for high-throughput, low-latency networking, which is particularly useful in massively parallel computer clusters. RDMA enables zero-copy networking by enabling network adapters to transfer data directly from the line to application memory or directly from application memory to the line, thus eliminating the need to copy data between application memory and the operating system's data buffers. Such transfers do not require work from the CPU, cache, or context switching, and the transfer continues in parallel with other system operations. This reduces latency in message transmission."

[0011] However, Wikipedia adds "This strategy presents several issues related to the fact that the target node is not notified of the completion of the request (one-sided communication)."

[0012] The use of network offloading in data centers is described in particular here: Mellanox.com / files / doc-2020 / sb-melanox-nvme-snap.pdf.

[0013] A socket typically includes the endpoints (typically an IP address and a port number) of a bidirectional communication link between first and second programs running on a network. A socket can be bound to a port number that enables the TCP layer to identify the application to which data will be sent.

[0014] NVMe / TCP is an example of a block storage protocol. NVMe is an example protocol; however, its reference in this document is not intended to be restrictive, and more generally, any suitable local or PCI storage protocol can be used instead.

[0015] Transmission Control Protocol / Internet Protocol (TCP / IP) is an example of a networking protocol that can be used by multiple computers to communicate with each other. Summary of the Invention

[0016] Some embodiments attempt to reduce the number of copies of the storage hierarchy buffer.

[0017] Some embodiments seek to provide a single device (e.g.). The card is exposed to multiple PCIe buses, where software is configured to map memory from one PCIe bus to be accessed by different PCIe buses, for example, using cross-PCIe mapping and / or transactions. A memory key can be used to describe the buffer (memory); the key can be defined to represent, for example, cross-PCIe (GVMI) mapping, as described herein. However, in PCIe NTB (non-transparent bridge) embodiments, a memory key is not required. Instead, conventional RDMA uses a pair of HW devices (HW devices) that define the 'RDMA connection'. (Cards or other devices) to cross PCIe boundaries. Each device may only handle its local PCIe buffer, but RDMA connections can transfer buffers between these devices.

[0018] Some embodiments attempt to provide cross-bus memory mapping.

[0019] More generally, some embodiments attempt to improve IOPS (IO operations per second) and / or reduce latency.

[0020] The scope of the present invention particularly includes the following embodiments:

[0021] Example 1. A computerized system for efficient interaction between a host and a second operating system, the host having a first operating system, the system comprising:

[0022] A subsystem, such as on the second operating system, typically retrieves data directly from a buffer local to the host, wherein the system is operable to map memory from a bus that may be associated with the first operating system to a different bus that may be associated with the second operating system, and typically accesses the memory from the different bus, thereby simulating a connection between the first and second operating systems, for example, through cross-bus memory mapping.

[0023] Example 2. A system according to any of the foregoing embodiments, wherein the first operating system has a PCI-type connection to the second operating system comprising multiple layers.

[0024] Example 3. A system according to any of the foregoing embodiments, wherein the subsystem includes a network stack.

[0025] Example 4. A system according to any of the foregoing embodiments, wherein the network stack is configured to recognize a buffer outside the memory of the second operating system itself, the buffer allowing software on the second operating system to access the buffer from the first operating system.

[0026] Example 5. The system according to any of the foregoing embodiments, wherein the host-local buffer resides on the host.

[0027] Example 6. A system according to any of the foregoing embodiments, wherein the system further includes an application on the host, the application being:

[0028] Open the storage on the emulated NVMe to provide the opened file;

[0029] Allocate a memory block in local memory and write data, referred to as buffer B, into the memory block;

[0030] Write to the opened file, thereby defining the write call; and

[0031] A kernel driver is provided that handles the write call and prepares the NVMe package with the buffer B.

[0032] Example 7. A system according to any of the foregoing embodiments, wherein the system includes NVMe emulation software and a TCP software stack that cooperate to provide cross-bus mapping by allowing cross-device memory key transfer through multiple software layers, thereby allowing software on the second operating system to access the host-local buffer from the first operating system.

[0033] Example 8. A system according to any of the foregoing embodiments, wherein the first operating system is further configured to control operations, thereby allowing the cross-device memory key transfer through the plurality of software layers, wherein the operations include:

[0034] Operation 25. The host's NVMe driver C publishes a request to buffer B to the NVMe emulation device D managed by the DPU, and the NVMe emulation device D exposes the NVMe device (emulated by D) to the host;

[0035] Operation 30. The NVMe emulation device D notifies the NVMe emulation device D's manager E on the DPU that the NVMe request is ready and that the manager E acquires a copy of the NVMe request to the manager E's local storage;

[0036] Operation 35. Manager E, for example, understands the NVMe request reference buffer B by accessing a copy of the NVMe request on Manager E's local storage, the buffer B being part of the client host storage and having a different PCIe bus or function;

[0037] Operation 40. Manager E initiates the new request with the attribute of performing a new request on the NVMe-TCP initiator driver F running on the DPU, and the new request or its attributes include local request information and the contents of a remote buffer (B);

[0038] Operation 50. The NVMe-TCP initiator driver F maintains a connection to the target's TCP socket via the TCP user-space stack G on the DPU;

[0039] Operation 53. The NVMe-TCP initiator driver F passes the NVMe-TCP format header of the request in the DPU local memory to the VMA TCP user space stack G, and the header is passed from F to G via the BSD send API;

[0040] Operation 57. The NVMe-TCP initiator driver F passes the contents of buffer B to the VMA TCP user space stack G, and the contents of buffer B are passed from G to G via the extended send API;

[0041] Operation 60. The TCP user space stack G manages the TCP stack and the HW send queue to send the DPU memory stream portion and the host memory stream portion to the local PCIe function bus without copying buffer B or one or more of the memory stream portions;

[0042] Operation 70. The NVMe-TCP initiator driver F obtains the NVMe-TCP response from the target regarding the sent NVMe-TCP request;

[0043] Operation 75. In response, the NVMe-TCP initiator driver F releases buffer B as the acknowledged relevant TCP stream portion in the TCP user space stack G.

[0044] Example 9. The system according to any of the foregoing embodiments, wherein the memory key includes an RDMA key.

[0045] Example 10. The system according to any of the foregoing embodiments, wherein the memory key includes a DMA key.

[0046] Example 11. A method for efficient interaction between a host and a remote target, the method comprising:

[0047] Provides a host with a first operating system and a remote target with a second operating system, the second operating system having a PCI-type connection to the first operating system; and

[0048] Sharing at least one buffer residing on the host with the remote target includes providing the target with zero-copy access to the at least one buffer, thereby providing double zero-copy across multiple operating systems.

[0049] Example 12. The method according to any of the preceding embodiments, wherein at least one of the host and the target uses a non-transparent bridge (NTB) for inter-domain communication via a PCIe interface.

[0050] Example 13. The method according to any of the preceding embodiments, wherein an I / O processing unit (IPU) of an array of ARM processor cores is deployed between the host and the network, and wherein SNAPTCP storage on the ARM processor core uses at least one XGVMI memory key to zero-copy PCIe access from the host.

[0051] Example 14. The method according to any of the preceding embodiments, wherein a first copy operation occurs from the host to the IPU, and a second copy occurs from the CPU memory to the TCP stack.

[0052] Example 15. The method according to any of the foregoing embodiments, wherein the method includes the following operations to increase IPOS and / or reduce latency and / or reduce CPU utilization by reducing the number of copies of the storage hierarchy buffer:

[0053] Operation 10. The application on the client host prepares a remote data buffer B on the host's local memory;

[0054] Operation 15. The host application requests the NVMe driver C on the client host to write the remote buffer B to storage;

[0055] Operation 20. The NVMe driver C on the client host prepares an NVMe request pointing to buffer B;

[0056] Operation 25. The host's NVMe driver C publishes the request pointing to buffer B to the NVMe emulation device D managed by the DPU, and the NVMe emulation device D exposes the NVMe device to the client host, wherein the NVMe emulation device D emulates the NVMe device;

[0057] Operation 30. The NVMe emulation device D notifies the NVMe emulation device D's manager E on the DPU that the NVMe request is ready, and the manager E obtains a copy of the NVMe request to the manager E's local memory;

[0058] Operation 35. Manager E understands that the NVMe request reference buffer B is part of the client host memory and therefore has a different PCIe bus or function;

[0059] Operation 40. Manager E initiates the new request with the attribute of performing a new request on the NVMe-TCP initiator driver F running on the DPU, and the new request or its attributes include local request information and the contents of the remote data buffer B;

[0060] Operation 50. The NVMe-TCP initiator driver F maintains a TCP socket connected to the target via the VMA TCP user space stack G on the DPU;

[0061] Operation 53. The NVMe-TCP initiator driver F passes the NVMe-TCP formatted header of the request in the DPU local memory to the VMA TCP user space stack G, and the header may be passed from the NVMe-TCP initiator driver F to the VMA TCP user space stack G via the BSD send() API;

[0062] Operation 57. The NVMe-TCP initiator driver F passes the contents of buffer B to the VMA TCP user space stack G, and the contents of buffer B are passed from the NVMe-TCP initiator driver F to the VMA TCP user space stack G via the extended send API;

[0063] Operation 60. The TCP user space stack G manages the TCP stack and HW send queue to send the DPU memory stream portion and host memory stream portion to the local PCIe function bus without copying buffer B or one or more of the memory stream portions;

[0064] Operation 70. The NVMe-TCP initiator driver F obtains from the target an NVMe-TCP response regarding the sent NVMe-TCP request; and

[0065] Operation 75. In response, the NVMe-TCP initiator driver F releases buffer B as the acknowledged relevant TCP stream portion in the VMA TCP user space stack G.

[0066] Example 16. The method according to any of the preceding embodiments, wherein the IPU includes a VMA, and wherein the network stack uses a memory key in addition to pointers to identify a buffer storing data to be sent over the network.

[0067] Example 17. The method according to any of the preceding embodiments, wherein the network stack directly extracts data to be sent over the network from the buffer residing on the host in a different PCIe domain.

[0068] Example 18. The method according to any of the preceding embodiments, wherein the remote target is connected to the host via a network unknown to the host.

[0069] Example 19. A system according to any of the foregoing embodiments, wherein the NTB (non-transparent bridge) is used to allow software on the second operating system to access a buffer local to the host from the first operating system.

[0070] Example 20. The system according to any of the foregoing embodiments, wherein the NTB (non-transparent bridge) is used by a VMA on the second operating system.

[0071] Example 21. A system according to any of the foregoing embodiments, wherein the network stack is configured to use a memory key instead of a pointer to identify the buffer.

[0072] Example 22. A system according to any of the preceding embodiments, wherein the NVMe-TCP initiator driver passes the contents of the buffer to the TCP user space stack in the network stack.

[0073] Example 23. The system according to any of the foregoing embodiments, wherein the second operating system includes a plurality of ARM CPUs.

[0074] Example 24. A system according to any of the foregoing embodiments, wherein ARM-based NVMe-TCP emulation software passes the buffer to a TCP socket in the VMA.

[0075] Example 25. A system according to any of the foregoing embodiments, wherein the second operating system directly extracts data to be sent over the network from a buffer residing on the host in a different PCIe domain.

[0076] Example 26. The system according to any of the foregoing embodiments, wherein the network stack includes a VMA. Attached Figure Description

[0077] Figure 1 This is a simplified block diagram illustrating a system for cross-bus memory mapping provided according to an embodiment of the present invention.

[0078] Figures 2 to 5 It is a table that can be used to create memory keys. Detailed Implementation

[0079] As used herein, the term “memory key” (Mkey) is intended to include a context with pointers and additional attributes, such as additional attributes of the pointers, which describe at least one of the following: how the memory pointer is accessed (e.g., read, write, atomic, all), which or which entities can access the pointer (e.g., protection key), what transformations can be performed to provide a virtual-to-physical mapping, and cross-GVMI information, as described herein.

[0080] It is understandable that pointers are known in the field of computer programming, for example, as described in the Wikipedia entry for "Pointer (computer programming)," and that the CPU (and other peripherals) can process data in memory space (if they are in registers or pointed to by pointers). Memory keys are described, for example, in Mellanox.com / related-docs / prod_software / Mellanox_OFED_Linux_User_Manual_v4_3.pdf.

[0081] MKEYs and their creation are described, for example, in 9ellanox.com / related-docs / user_manuals / Ethernet_Adapters_Programming_Manual.pdf. For instance, the CREATE_MKEY command described in that manual generates a new MKEY. The context of the new MKEY object can be obtained from the input mailbox. Typically, the hardware provides the MKEY index as an output parameter. This index can be used as a handle by software when accessing the object.

[0082] A VHCA is a Virtual Host Channel Adapter. VHCA and VHCA ID are described, in particular, in the commonly owned U.S. Patent US9397960B2, the disclosure of which is incorporated herein by reference. A multi-context single-logic virtual host channel adapter is described in that Cisco patent: https: / / patents.google.com / patent / US20080123672, the disclosure of which is incorporated herein by reference. It is known to have multiple Virtual Functions (VFs), each presenting itself as an independent VHCA to a host, and a network unaware of the VHCAs can only observe a single HCA. Methods for sharing the same physical port among multiple VHCAs are known and described, for example, in download.lenovo.com / servers / mig / 2020 / 05 / 28 / 21936 / mlnx-lnvgy_dd_nic_cx.ib-5.0-2.1.8.0-0_rhe17_x86-64.pdf (page 284).

[0083] The term "GVMI" (also known as Guest Virtual Machine Identifier) ​​typically refers to the hardware architecture; the GVMI is usually the name of the guest operating system of the virtual machine. An example GVMI is vhca_id (also known as "VHCA ID") or the VHCA identifier.

[0084] The term "cross-GMVI Mkey" refers to something that exists in, for example... The hardware architecture in X6Dx is described at the following www link: nvidia.com / en-us / networking / ethernet / Mellano-6-dx. The terms XGVMI, cross-GVMI, and cross-domain GVMI are generally used interchangeably in this document. It can be understood that network adapters, such as... The X6Dx features a hardware portion that supports cross-GVMI Mkey operations, combined with firmware running on the internal core. This firmware typically exposes a protected command interface documented by a Programmer's Reference Manual (aka PRM), which is available here: Mellanox.com / related-docs / user_manuals / Ethernet_Adapters_Programming_Manual.pdf.

[0085] However, it should be understood that any suitable Ethernet network adapter driver that supports the described cross-GVMI MKEY can be used. The firmware code can take a command and map that command to the HW configuration. This is only an example implementation, which provides flexibility in keeping the command the same, even as new hardware versions come out. In the device API implemented by the above firmware, this can be shown as: "CREATE_MKEY with access_mode=0x6:CROSSING_VHCA_MKEY".

[0086] Figure 1 A system for cross-bus memory mapping according to an embodiment of the present invention is shown, which generally facilitates storage NVMe emulation on NVMe-TCP acceleration. Figure 1 Systems typically include all or any suitable subset of the following:

[0087] Client applications (also known as "performance applications" or "perf") run on the host machine (also known as the client host). It's important to understand that... Figure 1 The software residing on the host operating system typically controls the operations described below 10-75;

[0088] DPU (e.g., )

[0089] The NVMe driver on the client host (also known as the host's NVMe driver C);

[0090] The NVMe-TCP initiator driver F runs on the DPU;

[0091] VMA TCP userspace stack G on the DPU;

[0092] VHCA;

[0093] GVMI A and B are typically implemented in hardware, and either or both can be replaced by VHCA; typically, GVMI A is defined as a network device. GVMI B typically includes an NVMe emulation (PCIe, for example) device that exposes the NVMe device to the client host and manages it via the DPU;

[0094] Software on a second operating system (e.g., ), used to access the host-local buffer from the first operating system;

[0095] TCP sockets in VMA; and

[0096] The NVMe-TCP emulation software passes buffer B to the VMA TCP socket, which can... Implemented or as accomplish.

[0097] A cross-GVMI memory key (also known as a cross-GVMI-Mkey) is configured to retrieve a buffer from one GVMI (e.g., GVMI A) and use the same buffer on the other side as the second GVMI (e.g., GVMI B).

[0098] A DPU is a possible example of a data processor, which is typically a system-on-a-chip (SoC) and usually includes:

[0099] A software-programmable multi-core CPU, which may have an ARM architecture, is typically coupled to other SoC components, and / or

[0100] High-performance network interfaces for parsing, processing, and transmitting data to, for example, GPUs and / or CPUs at the remaining speed of the network (aka line rate); and / or

[0101] A programmable acceleration engine that offloads applications (e.g., AI) and / or machine learning and / or security and / or telecommunications and / or storage, thereby improving application performance.

[0102] It should be understood that some embodiments described herein are beneficial for reducing the NVMe payload transferred from host (e.g., x86) memory to NVMe emulation software from the DPU CPU core (e.g., ...). The wait time and CPU usage caused by the replication overhead of managed TCP packet payloads.

[0103] Typically, NVMe emulation software (e.g., The network interface card (NIC) hardware is configured to send at least one NVMe-TCP packet, such that the TCP header is collected from the local DPU memory, while the TCP payload is collected directly from the host (x86) memory on the remote PCIe bus. This excludes or skips up to two skipped copies and hierarchical buffers.

[0104] A particular advantage of certain embodiments is that the network interface card (NIC) hardware typically collects a single packet from multiple memory addresses that extend across multiple PCIe buses. Software is used to translate and / or map and / or manage these memory addresses from their respective PCIe buses, and the hardware is configured accordingly to send such packets.

[0105] The I / O processing unit (IPU) is a system-on-a-chip (SoC), which includes a SmartNIC, for example, DX, an ARM processor core array, features a high-performance memory interface and flexible processing capabilities. SmartNICs may include a hardware offload engine with data cryptography in IPsec and inline TLS motion, and / or network virtualization, and / or RDMA over aggregated Ethernet (RoCE), and / or on-structure NVMe (NVMe-oF) memory acceleration, as described online at: mellanox.com / news / press_release / mellanox-introduces-revolutionary-connectx-6-dx-and-bluefield-2-secure-cloud-smartnics-and-io. ARM processors are CPUs based on the Advanced RISC Machine (ARM) RISC (Reduced Instruction Set Computer) architecture, as described online at: whatis.techtarget.com / definition / ARM-processor#:~:text=An%20ARM%20processor%20is%20one,bit%20RISC%20multi%2Dcore%20processors.

[0106] Figure 1 The system can be used in conjunction with known technologies, such as and / or VMA and / or TCP sockets and / or libibverbs and / or But these are not intended to be restrictions. For example, Figure 1 The second operating system may include multiple ARM CPUs, arrays of ARM processor cores (such as those described herein), as described in this document. IPU, also known as (ARM), or alternatively, it may not be ARM-based at all, and can be said to be x86 or based on any other suitable CPU architecture. Mellanox NVMe Software-defined network acceleration (SDN) is an example of a framework or technology that implements hardware-accelerated virtualization of NVMe storage, such as by emulating block storage devices to the host. For example, an NVMe driver on the PCIe bus can be emulated, making the networked storage appear as a local NVMe SSD. The host OS / hypervisor can utilize its standard NVMe driver, unaware that the communication is handled by NVMe. Instead of being terminated by the physical driver, it can then be terminated via NVMe. The data is applied to the storage target via a network, such as over Ethernet or a wireless bandwidth protocol. Alternatively, memory virtualization in the hardware and / or optimization for CAPEX and OPEX savings in storage resources can be implemented and / or configured to free up computer node CPU resources by offloading both networking and storage data paths, and preferably any other suitable technology based on any network protocol can be integrated into any storage solution.

[0107] Regarding [Copy #2], it should be noted that the online link: kernel.org / doc / html / v4.15 / networking / msg_zerocopy.html provides the Linux socket definition for the zero-copy flag. A TCP stack implementation (VMA) can add support for send with the MSG_ZEROCOPY flag. Implementations of send() calls with the MSG_ZERCOOPY flag can use the application buffer pointer as a reference without copying it into an intermediate buffer. The TCP stack manages the reference buffer pointer used to send TCP packets, and once a TCP ACK covering the entire send buffer size is received, the TCP stack can simply release the reference on the buffer. The TCP socket owner (SPDK initiator) can call send with MSG_ZEROCOPY when supported by a TCP stack implementation (VMA). The implementation of send() by the caller with MSG_ZERCOOPY can set aside the send buffer until it receives an indication that the TCP stack has finished sending and receives an acknowledgment.

[0108] Regarding [copy #1], it may include Zerocopy (For example, host-> This allows APIs to be added to the TCP stack (e.g., VMA) to provide a cross-GVMI Mkey, enabling the VMA to later detect the marked physical address (aka "PA") as host memory, and this cross-domain GMVI memory key can be used for PCIe collection in the HW. Preparing a cross-GVMI Mkey... The cross-GVMI Mkey can be passed to the VMA during TCP socket initialization. Signaling the TCP buffer as needed can be accomplished by appropriately marking the external host buffer pointer. "External host memory" across GVMI Mkey between (e.g.) VMA. For example, the MSB of the physical address can be set accordingly. Logic can be added to (e.g.) VMA to detect the PA of the flag as a host external memory buffer, and the Mkey across GMVI VMA can be used to set hardware with a data buffer SGE on the original QP (SQ / TIS) with the GVMI Mkey.

[0109] For simplicity, this manual uses examples to refer to SNAP. TM However, such a reference is not intended to be restrictive, and more generally, any storage device emulator (e.g., smartNIC service) can be used to emulate NVMe (or blank-blk) storage PCIe devices to a host by exposing the storage PCIe device interface to the host (or enabling the host to access the storage PCIe device).

[0110] The following operating methods are provided, individually or in conjunction with... Figure 1 The system combination, or more generally the combination with all or any subset of the following:

[0111] A. The application on the client host (also known as host application A)

[0112] B. Data buffer on the client host's memory (also known as buffer B)

[0113] C. NVMe driver on the client host (also known as host NVMe driver C)

[0114] D. NVMe emulation device (exposing NVMe devices to client hosts and managing them via DPU)

[0115] E. NVMe emulation device manager on the DPU, for example

[0116] F. NVMe-TCP Initiator Driver F, running on, for example, a DPU.

[0117] G.VMA (for example) TCP user space stack G, which can be on the DPU.

[0118] This method typically includes all or any subset of the following operations in appropriate order, for example, as follows:

[0119] Operation 10. The application on the client host prepares remote data buffer B on the host's local storage;

[0120] Operation 15. The host application requests the NVMe driver C on the client host to write the remote buffer B to storage;

[0121] Operation 20. The host's NVMe driver C prepares an NVMe request pointing to buffer B;

[0122] Operation 25. The host's NVMe driver C publishes a request to buffer B to the NVMe emulation device D managed by the DPU, and the NVMe emulation device D exposes the NVMe device to the client host, emulating the NVMe device;

[0123] Operation 30. The NVMe emulation device D notifies the NVMe emulation device D's manager E on the DPU that the NVMe request is ready and that the manager E acquires a copy of the NVMe request to the manager E's local storage;

[0124] Operation 35. Manager E, for example, understands the NVMe request reference buffer B by accessing a copy of the NVMe request on Manager E's local storage, which is part of the client host storage and thus has a different PCIe bus or function;

[0125] Operation 40. Manager E initiates the new request on the NVMe-TCP initiator driver F running on the DPU, which has the attribute of performing a new request, the new request or its attributes including local request information and the contents of remote data buffer B;

[0126] Operation 50. The NVMe-TCP initiator driver F maintains a connection to the target TCP socket via the VMA (e.g.) TCP user-space stack G on the DPU;

[0127] Operation 53. The NVMe-TCP initiator driver F passes the NVMe-TCP formatted header of the request from the DPU local memory to the VMA TCP user space stack G. This header can be passed from the NVMe-TCP initiator driver F to the VMA TCP user space stack G via the BSD send() API.

[0128] Operation 57. The NVMe-TCP initiator driver F passes the contents of buffer B to the VMA TCP user-space stack G. The contents of buffer B can be passed from the NVMe-TCP initiator driver F to the VMA TCP user-space stack G via the extended send API; therefore, VMA becomes able to use the memory key instead of the pointer ID (identifying) buffer.

[0129] Operation 60.VMA TCP user space stack G manages the TCP stack and HW send queue to send DPU memory stream portions and host memory stream portions to the local PCIe function bus without copying buffer B or memory stream portions;

[0130] Operation 70. The NVMe-TCP initiator driver F obtains from the target the NVMe-TCP response regarding the sent NVMe-TCP request; and

[0131] Operation 75. In response, the NVMe-TCP initiator driver F releases buffer B as the acknowledged (ACK) portion of the relevant TCP stream in the VMA TCP user space stack G.

[0132] Many variations are possible relative to the specific embodiments described herein by way of example only. Only a few examples are given:

[0133] The systems and methods described here can be integrated into any suitable system, such as SNAP, NVIDIA Mellanox, etc. Alternatively, VMA. However, it should be understood that the use of VMA is only one possible implementation. VMA is one possible TCP / IP stack solution, but alternatively, in non-VMA solutions, a TCP / IP stack can be provided that is modified, for example, to handle cross-GVMI Mkey as described herein. It should be recognized that VMA (or Mellanox Messaging Accelerator) is an example of a network stack; other examples include, but are not limited to, the Linux TCP stack, mTCP, Seastar, and other network stacks described online, such as on the DPDK ecosystem page, which is available online at dpdk.org / ecosystem / .

[0134] Furthermore, the present invention is not limited to PCI type connections (e.g., dedicated to PCI or PCIe connections) and can be extended to buses other than PCI and PCIe.

[0135] Furthermore, any suitable method can be used to create the memory key. For example, Figure 2 , Figure 3 , Figure 4 , Figure 5 Tables 226-229, as shown, are in mellanox.com / related-docs / user_manuals / Ethernet_Adapters_Programming_Manual.pdf. These respectively represent the example CREATE_MKEY input structure layout ( Figure 2 Example CREATE_MKEY input structure field description ( Figure 3 Example CREATE_MKEY output structure layout ( Figure 4) and example CREATE_MKEY output structure field descriptions ( Figure 5 ).

[0136] Furthermore, the NTB (Non-Transparent Bridge)-based approach can replace the use of cross-domain GVMI memory keys (aka "cross-GVMIMkey"), as described herein, for example by providing software on a second operating system (e.g., [missing information]) configured to allow VMA to use NTB (e.g., as in a cross-GVMIMkey solution). Software that accesses the host's local buffer from the first operating system.

[0137] Furthermore, as described, according to certain embodiments, VMA directly extracts data to be sent over the network from a buffer residing on a host in a different PCI or PCIe domain. Any suitable implementation can be used to enable VMA to do this. For storage-related solutions, the application on the host can perform all or any suitable subset of the following:

[0138] Phase a. Open the storage on the emulated NVMe (e.g., open (“path”)); and / or

[0139] Phase b. Allocate local memory (e.g., malloc()) and write its data to the allocated memory block [e.g., as described herein with reference to operation 10]; and / or

[0140] Stage c.Write() to the opened file [e.g., as described herein with reference to Operation 15]; and / or

[0141] Stage d. The kernel driver can handle write() calls and can prepare an NVMe package with buffer B, typically from stage b, for example, as described in reference operation 20 of this document.

[0142] As used herein, the term opaque is the opposite of a “transparent” connection. For example, a PCIe NTB (non-transparent bridge) can be used to provide connectivity between multiple CPUs, GPUs, and FPGAs. When using a PCIe (transparent) bridge, all PCIe buses (all the way to all endpoints) are visible to a given root complex. Conversely, a non-transparent bridge forwards PCIe traffic between separate PCIe buses and each root complex RC, treating the NTB (non-transparent bridge) as an endpoint device, while devices on the root complex and the other side of the bridge are not visible to the RC, for example, as described online at: missinglinkelectronics.com / devzone / index.php / fpga-debug-ntb.

[0143] VMA (Voltaire Messaging Accelerator) is an example data acceleration technology. NVIDIA's VMA (released as open source) enhances the performance of message-based and streaming applications across a wide range of industries, such as high-frequency trading (HFT) platforms and Web 2.0 clusters. VMA exposes a standard socket API with a kernel-bypass architecture, enabling user-space networking for multicast, UDP unicast, and TCP streaming. VMA can be used off-the-box, for example, due to built-in pre-configured profiles, for example, for latency and streaming. VMA is described at the following online locations: docs.mellanox.com / category / vma and docs.mellanox.com / display / VMAv911 / Introduction+to+VMA; it should be understood that other acceleration technologies may include subsets of the features, characteristics, and capabilities provided by VMA.

[0144] All publications and patent documents mentioned in this specification, as well as the publications and patent documents directly or indirectly referenced therein, are incorporated herein by reference, unless subject to abandonment or denial.

[0145] It should be understood that, if desired, the software components of this invention can be implemented in the form of ROM (Read-Only Memory). If desired, the software components can typically be implemented in firmware or hardware using conventional techniques. It should also be understood that the software components can be instantiated, for example, as a computer program product or on a tangible medium. In some cases, it is possible to instantiate the software components as signals that can be interpreted by a suitable computer, although such instantiation may be excluded in certain embodiments of this invention.

[0146] It should be understood that, for clarity, the various features of the invention described in the context of individual embodiments may also be provided in combination in a single embodiment. Conversely, for brevity, the various features of the invention described in the context of individual embodiments may also be provided individually or in any suitable sub-combination.

[0147] Those skilled in the art will understand that the present invention is not limited to what has been specifically shown and described above. Rather, the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A computerized system for efficient interaction between a host and a second operating system, the host having a first operating system, the system comprising: A subsystem on the second operating system, the subsystem directly extracting data from the host-local buffer, wherein the system is operable to map memory from a bus associated with the first operating system to a different bus associated with the second operating system, and to access the memory from the different bus, thereby emulating the connection between the first operating system and the second operating system through cross-bus memory mapping to reduce the number of copy operations in memory transactions involving the memory; The subsystem includes NVMe emulation software and a TCP software stack that cooperate to provide cross-bus mapping by allowing the transfer of cross-device memory keys that identify the buffer through multiple software layers, thereby allowing software on the second operating system to access the buffer from the first operating system.

2. The system of claim 1, wherein the first operating system has a PCI-type connection to the second operating system comprising multiple layers.

3. The system of claim 1, wherein the host-local buffer resides on the host.

4. The system according to claim 1, further comprising an application program on the host computer, the application program being: Open the storage on the emulated NVMe to provide the opened file; Allocate a memory block in local memory and write the data in the buffer into the memory block; Write to the opened file, thereby defining the write call; as well as A kernel driver is provided that handles the write call and prepares the NVMe package with the buffer.

5. The system of claim 1, wherein the first operating system is further configured to control operations that allow the cross-device memory key transfer through the plurality of software layers, wherein the operations include: The host's NVMe driver publishes requests to the buffer to the NVMe emulation device managed by the DPU, and the NVMe emulation device exposes the NVMe device to the host. The NVMe emulation device notifies the NVMe emulation device manager on the DPU that the NVMe request is ready, and obtains a copy of the NVMe request on the local memory of the NVMe emulation device manager. The NVMe request references the buffer, which has a different PCIe bus or function, by accessing the copy of the NVMe request on the local memory through the manager of the NVMe emulation device. The new request, which has the attribute of performing a new request, is initiated by the manager of the NVMe emulation device on the NVMe-TCP initiator driver running on the DPU, wherein the new request or its attributes include local request information and the contents of the remote buffer; The TCP socket connection to the target is maintained via the TCP user-space stack on the DPU through the NVMe-TCP initiator driver; The NVMe-TCP format header of the new request in local memory is passed to the TCP userspace stack via the BSD Send API through the NVMe-TCP launcher driver; The contents of the buffer are passed to the TCP userspace stack via the extended send API through the NVMe-TCP launcher driver; The TCP user space stack and send queue are managed through the TCP user space stack to send the DPU memory stream portion and the host memory stream portion to the local PCIe function bus without copying the buffer and the memory stream portion; The NVMe-TCP initiator driver obtains the NVMe-TCP response from the target regarding the sent NVMe-TCP request. In response, the buffer is released by the NVMe-TCP initiator driver as a relevant TCP stream portion in the TCP user space stack.

6. The system of claim 1, wherein the cross-device memory key includes an RDMA key.

7. The system of claim 1, wherein the cross-device memory key includes a DMA key.

8. A method for efficient interaction between a host and a remote target, the method comprising: Provides a host with a first operating system and a remote target with a second operating system, the second operating system having a PCI-type connection to the first operating system; as well as Sharing at least one buffer residing on the host with the remote target includes providing the remote target with zero-copy access to the at least one buffer, thereby providing double zero-copy across the first and second operating systems to reduce the number of copy operations in memory transactions between the host and the remote target. This includes an I / O processing unit (IPU) of an array of ARM processor cores deployed between the host and the network, and wherein storage on the ARM processor cores uses at least one memory key that identifies the at least one buffer for zero-copy PCIe access from the host.

9. The method of claim 8, wherein at least one of the host and the remote target uses a non-transparent bridge (NTB) for inter-domain communication via a PCIe interface.

10. The method of claim 8, wherein the first copy operation occurs from the host to the IPU, and the second copy occurs from the CPU memory of the IPU to the TCP stack.

11. The method of claim 10, wherein the method comprises the following operations to increase IPOS and / or reduce latency and / or reduce CPU utilization by reducing the number of copies of the storage hierarchy buffer: The host's NVMe driver publishes requests to the buffer to the NVMe emulation device managed by the DPU, and the NVMe emulation device exposes the NVMe device to the host. The NVMe emulation device notifies the NVMe emulation device manager on the DPU that the NVMe request is ready, and obtains a copy of the NVMe request on the local memory of the NVMe emulation device manager. The NVMe request references the buffer, which has a different PCIe bus or function, by accessing the copy of the NVMe request on the local memory through the manager of the NVMe emulation device. The new request, which has the attribute of performing a new request, is initiated by the manager of the NVMe emulation device on the NVMe-TCP initiator driver running on the DPU, wherein the new request or its attributes include local request information and the contents of the remote buffer; The TCP socket connection to the target is maintained via the TCP user-space stack on the DPU through the NVMe-TCP initiator driver; The NVMe-TCP format header of the new request in local memory is passed to the TCP userspace stack via the BSD Send API through the NVMe-TCP launcher driver; The contents of the buffer are passed to the TCP userspace stack via the extended send API through the NVMe-TCP launcher driver; The TCP user space stack and send queue are managed through the TCP user space stack to send the DPU memory stream portion and the host memory stream portion to the local PCIe function bus without copying the buffer and the memory stream portion; The NVMe-TCP initiator driver obtains the NVMe-TCP response from the target regarding the sent NVMe-TCP request. In response, the buffer is released by the NVMe-TCP initiator driver as a relevant TCP stream portion in the TCP user space stack.

12. The method of claim 8, wherein the IPU further uses a pointer to identify the buffer.

13. The method of claim 12, wherein the IPU extracts data directly from the buffer residing on the host in a different PCIe domain.

14. The method of claim 8, wherein the remote target is connected to the host via a network unknown to the host.

15. The system of claim 1, wherein the non-transparent bridge NTB is used to allow software on the second operating system to access the buffer from the first operating system.

16. The system of claim 1, wherein the NVMe-TCP initiator driver passes the contents of the buffer to the TCP user space stack in the TCP stack.

17. The system of claim 16, wherein the second operating system comprises a plurality of ARM CPUs.

18. The system of claim 17, wherein the NVMe-TCP emulation software is ARM-based and passes the buffer to the TCP socket in the VMA.

19. The system of claim 2, wherein the host is in a different PCIe domain.

Images