Apparatus, method, and computer program product for efficient software-defined network acceleration processing using host-local storage

Abstract

The present disclosure relates to apparatus, methods, and computer program products for efficient software-defined network acceleration processing using storage local to a host. A computerized system operating in conjunction with a computerized apparatus and a fabric target service in data communication with the computerized apparatus, the system comprising functionality resident on the computerized apparatus and functionality resident on the fabric target service, when operating in combination, enable the computerized apparatus to coordinate access to data.

Description

Technical Field

[0001] This invention relates generally to hardware storage devices, and more specifically to networked hardware storage devices. Background Technology

[0002] Networked storage allows multiple application / database servers to share storage via Storage Area Network Protocol (SAN) on a single Network Attached Storage (NAS) server or on a block storage controller. Various storage networking protocols are known to support both NAS and SAN systems. The same storage enclosure can support distributed file (e.g., SAN) and block storage protocols.

[0003] For example, DMA (Direct Memory Access) between GPU memory and storage devices such as NVMe storage drives is known, and is described, for example, here: news.developer.nvidia.com / gpu-direct-storage-early-access /

[0004] DMA transfers are also described here: docs.nvidia.com / cuda / gpudirect-rdma / index.html#standard-dma-transfer-example-sequence

[0005] DPU (Data Processing Unit) and Direct Memory Access (DMA) typically include a platform that includes a networking interface (such as...). of The DPU includes a programmable processing module (such as an ARM core) that provides the processing capabilities of the DPU and can be programmable. Versions 1, 2, and 3 are all examples of existing DPUs.

[0006] The DPU can be used as a standalone embedded processor or integrated into a network interface controller. For example, the DPU can be incorporated into a SmartNIC, a network interface controller used in servers such as "next-generation servers," which are a class of servers characterized by increased processor speed, enhanced management features, and higher energy efficiency compared to other servers.

[0007] SmartNIC technology is described here: blog.melinox.com / 2018 / 08 / defining-smartnic /

[0008] The Storage Performance Development Kit (SPDK) is a toolkit for developers that provides building blocks for scalable, high-performance storage applications. SPDK stands for "NVMe over Fabrics target," which provides user-space applications for block devices via "fabrics" such as Ethernet, Infiniband, or Fibre Channel.

[0009] The structural NVMe specification defines subsystems that can be exported through different transports; note that both RDMA and TCP transports are supported by SPDK. The SPDK specification refers to the software that exports these subsystems as a "target" (a term used for iSCSI) and the client of the target as a "host." It should be recognized that the Linux kernel also implements NVMe-oF "targets" and "hosts."

[0010] As described online here (nvmexpress.org / about), NVM Express is "an open collection of standards and information that fully demonstrates the benefits of non-volatile memory in all types of computing environments, from mobile to data centers. The original NVM Express Working Group... is the consortium responsible for developing the NVM Express specifications... The NVM Express standards include:

[0011] • NVM Express (NVMe) specification – A register interface and command set for PCIe SSDs with industry-standard software available for many operating systems. NVMe is widely considered the de facto industry standard for PCIe SSDs.

[0012] • NVMe Management Interface (NVMe-MI) Specification - The command set and architecture for out-of-band management of NVM Express storage (e.g., using BMC to discover, monitor, and update NVMe devices).

[0013] • Architectural NVMe (NVMe over Fabrics, NVMe-oF) specification – an extension of NVM Express that enables the NVM Express command set to tunnel over additional transports outside of the PCIe architecture. Architectural NVMe technology benefits from scaling high-performance storage architectures at scale in the world's largest data centers by allowing the same protocol to be extended across various networking interfaces.

[0014] Ampere is... Examples of existing technology graphics processing unit (GPU) microarchitectures developed and included of -2X DPU.

[0015] The architectural characteristics of a GPU may include all or any subset of the following characteristics, all of which characterize Ampere:

[0016] - The A100 has a CUDA computing power of 8.0, while the GeForce 30 series has a CUDA computing power of 8.6.

[0017] - TSMC's 7nm FinFET process for A100

[0018] - Samsung's custom version of the GeForce 30 series using 8nm process (8N)

[0019] - Features third-generation tensor core support and sparse acceleration for FP16, bfloat16, TensorFloat-32 (TF32), and FP64.

[0020] - Second-generation ray tracing core; GeForce 30 series concurrent ray tracing, shading, and computation.

[0021] High-bandwidth memory 2 (HBM2) on the A100 40GB and A100 80GB

[0022] - GDDR6X memory for GeForce RTX 3090 and 3080

[0023] - Dual FP32 cores per SM on the GA10x GPU

[0024] - NVLink 3.0 has a throughput of 50 Gbit / s per pair.

[0025] - PCI Express 4.0 with SR-IOV support (SR-IOV is reserved only for A100)

[0026] -Supports multi-instance GPU (MIG) virtualization and GPU partitioning features in A100 processors with up to seven instances.

[0027] - Pure video feature set K with AV1 hardware decoding for GeForce 30 series and feature set J for A100.

[0028] -5NVDEC for A100

[0029] - New hardware-based 5-core JPEG decoding (NVJPG) with YUV420, YUV422, YUV444, YUV400, and RGBA.

[0030] This article describes, online, prior art that is useful in conjunction with certain embodiments: devops.com / the-nvmf-boogie-kubernetes-and-nvmf-full-rock.

[0031] NVM Express Revision 1.2.1 and its earlier revisions are documents defining a register-level interface for host software to communicate with the non-volatile memory subsystem via PCI Express (NVMe via PCIe). The specification available online at nvmexpress.org / wp-content / uploads / NVMe-over-Fabrics-1_0a-2018.07.23-Ratified.pdf defines extensions to NVMe that enable operation on other interconnects (structural NVMe).

[0032] All publications and patent documents mentioned in this specification, and all publications and patent documents directly or indirectly referenced herein, are incorporated herein by reference, except in subject matter disclaimers or denials. Summary of the Invention

[0033] According to some embodiments, the host accesses data via an emulated storage device, which may be partially implemented in software (e.g., DPU software). The emulated storage device enables the location of at least a portion of the data accessed by the host via the emulated storage device interface, and at least a portion of this data is available in a separate attached storage device among a group of storage devices locally attached to the host but not attached to the DPU.

[0034] In the DPU software, the simulated storage device accesses the host-attached (i.e., attached to the host) storage device, assisted by the architecture target service (on or off the host), thereby generating a set of DMAs comprising 2-3 DMAs (also known as DMA transfers, or direct memory access transfers). Alternatively, a single direct memory access transfer can be used, as described herein.

[0035] The terms “target service,” “structure target service,” or “storage target service” as used herein are typically RDMA-based and intended to encompass software and / or hardware running on a storage target device in the context of network block storage or SAN (Storage Area Network). In SAN / network block storage terminology, “initiator” and “target” can encompass both sides of this network connection: the “initiator” can be a host that typically requests storage services and initiates storage requests, whether reads or writes; and the “target” can be a storage device that serves the initiator in sending reads and writes to the target.

[0036] In some embodiments, the SAN is internal between the DPU and the host connected to the DPU via PCIe, without involving network ports.

[0037] A particular advantage of certain embodiments is that instead of developing entirely different connections to enable the DPU to send requests to the host intended for use with a local driver, the embodiments herein use known services (modified as described herein), typically employing hardware offloading features to assist the host in acting as a storage target.

[0038] All or any subset of the following can characterize the solutions provided according to embodiments of the present invention:

[0039] a. The structural NVMe (NVMf) initiator in the SmartNIC is configured to identify (e.g., by handshaking with the target) or to understand whether a particular network NVMf subsystem target can handle NVMe requests (e.g., submitting queue entries, i.e., SQE) instead of NVMf requests.

[0040] b. In SmartNIc, the NVMf initiator can write SQEs and pass them to the NVMf target, for example, using new vendor-specific opcodes.

[0041] c. SmartNIC's storage stack is able to handle raw host addresses, split them into different partial requests, and send only the appropriate subset of them, rather than all of them, to the host's local physical NVMe device.

[0042] d. The NVMe-oF standard has been modified to increase the ability to handle network requests involving local system memory. Typically, functionality on an NVMe SQE target (e.g., software on the target) identifies certain requests as NVMe SQE rather than regular NVMe requests by pre-configuration of the subsystem, by using vendor-specific opcodes, or by providing another indication within the SQE itself.

[0043] The e.NVMf target software can skip any network data movement.

[0044] f.NVMf target software converts vendor-specific opcodes (if used) into standard opcodes.

[0045] The g.NVMf target software submits the resulting SQE to the physical NVMe drive.

[0046] Typically, vendor-specific NVMe opcodes are used to indicate data requests, such as those described herein. Vendor-specific READ / WRITE commands can be defined. In an NVMe implementation, when such a vendor-specific command arrives, the opcode is replaced with the original READ / WRITE before being directly committed to the disk, including bypassing the RDMA data in / outside the staging buffer.

[0047] According to some embodiments, the existing fields 'PSDT' and 'SGL identifier' within the SQE can be used to indicate that the SQE refers to host storage rather than the network by configuring them to values ​​that are not 'Keyed SGL block descriptor' (as required by a regular NVMf request (with a network address)).

[0048] At least the following embodiments are provided:

[0049] Example 1. A computerized system that operates in conjunction with a computerized device and a structural target service, the structural target service communicating data with the computerized device, the system comprising:

[0050] Functions residing on the computerized device; and

[0051] Functions residing on the structural target service

[0052] When combined, the function enables the computerized device to coordinate access to data.

[0053] Example 2. A computerized system according to any of the embodiments described herein, wherein the computerized system enables a host to access a emulated PCI storage device by using the computerized means for coordinating the execution of a host request arriving at an interface of the emulation, using fewer than three direct memory access (DMA) transfers, wherein the host request relates to data stored or to be stored on at least one hardware storage device locally attached to the host, wherein the data is transferred between the host’s original application buffer and the hardware storage device locally attached to the host.

[0054] Example 3. A computerized system according to any of the embodiments described herein, wherein the functions residing on the computerized device are used to perform at least one of the following:

[0055] a. Provide the simulated storage device to the host;

[0056] b. Obtain requests published on the emulated device from the host;

[0057] c. Parse the request;

[0058] d. Determine that one or more storage devices attached to the host are the source or destination of at least a portion of the data referenced in the request, and in response, determine that the one or more storage devices attached to the host should be involved in the request; and

[0059] e. Prepare at least one request having a local memory address pointing to the original buffer located in the host's memory, to be sent to the one or more hardware storage devices attached to the host.

[0060] Example 4. A computerized system according to any of the embodiments described herein, wherein the request having a local memory address is formatted against a network storage protocol, and wherein the function residing on the structure target service is used for:

[0061] a. Identify the request with a local memory address by recognizing a new opcode or indication within an existing opcode; and / or

[0062] b. Accordingly, using a pointer to the original buffer received in the request having a local memory address, a regular request formatted for the local storage protocol is generated; and / or

[0063] c. The regular request is published to the locally attached storage device managed by the local storage protocol, which will cause the locally attached storage device to initiate at least one DMA transfer of data between the host buffer and the internal flash memory.

[0064] Therefore, typically, the target service creates a “request” (also known as a “regular” request) with a local host storage address instead of a network address / multiple network addresses, which is later posted on the physical NVMe device.

[0065] Example 5. A computerized system according to any of the embodiments described herein, wherein the computerized apparatus includes: a storage device emulator that emulates a storage device on a PCIe bus; and the architecture target service, wherein the storage device emulator includes a driver emulator that emulates a driver on a PCIe bus.

[0066] Example 6. A computerized system according to any of the embodiments described herein, wherein the driver emulator includes an NVMe driver emulator that emulates an NVMe driver on a PCIe bus.

[0067] Example 7. A computerized system according to any of the embodiments described herein, wherein the NVMe driver emulator employs software-defined network acceleration processing.

[0068] Example 8. A computerized system according to any of the embodiments described herein, wherein the structure target service is provided by a host locally attached to a hardware storage device, and the structure target service further includes a structure target hardware offloading that performs at least some functions on behalf of the structure target service.

[0069] Example 9. A computerized system according to any of the embodiments described herein, wherein the hardware storage device includes a physical NVMe device and the emulated PCI storage device are in the same passthrough domain, and wherein the same address is used to describe the host's raw application buffer for both the physical NVMe device and the emulated PCI storage device.

[0070] Example 10. A computerized system according to any of the embodiments described herein, wherein the original application buffer is part of a virtual machine (VM) memory space, and wherein the physical NVMe device accesses the original application buffer by using PASID (Process Address Space ID) technology on the physical NVMe side.

[0071] Example 11. A computerized system according to any of the embodiments described herein, wherein the original application buffer is part of a virtual machine (VM) memory space, and wherein the physical NVMe device accesses the original application buffer by creating an Input / Output Memory Management Unit (IOMMU) domain comprising all memory domains of a plurality of virtual machines, deploying the physical NVMe device in the IOMMU domain, and translating at least one original address to an address that matches at least one of the memory domains of the plurality of virtual machines.

[0072] Example 12. A computerized system according to any of the embodiments described herein, wherein the original application buffer is part of a virtual machine (VM) memory space, and wherein the physical NVMe device accesses the original application buffer by using an ATS (Address Translation Service) to provide input / output memory management unit (IOMMU) translation of at least one address.

[0073] It is understood that, according to some embodiments, the emulated storage PCI device is passed to the virtual machine (VM), while the physically attached NVMe driver belongs to the host, not the VM.

[0074] Example 13. A computerized system according to any of the embodiments described herein, wherein the local storage protocol includes PCI storage protocols from the group consisting of: NVMe, Virtio-blk, Virtio-scsi, SCSI, SATA, SAS, and IDE.

[0075] Example 14. A computerized system according to any of the embodiments described herein, wherein the network storage protocol includes a block storage network storage protocol.

[0076] Example 15. A computerized system according to any of the embodiments described herein, wherein the block storage network storage protocol includes remote direct memory access (RDMA) network block storage protocols from the group consisting of NVMe-oF, iSER, and SRP.

[0077] Example 16. A method for enabling a host to access an emulated PCI storage device, the method comprising: employing computerized means to coordinate the execution of a host request arriving at the emulated interface, while using fewer than three direct memory access (DMA) transfers, wherein the host request relates to data stored or to be stored on at least one hardware storage device locally attached to the host.

[0078] Example 17. A method according to any of the embodiments described herein, wherein the host uses a plurality of direct memory access transfers to access the emulated PCI storage device, and wherein the access includes enabling the computerized device to coordinate access to the data.

[0079] Example 18. A method according to any of the embodiments described herein, wherein the data is stored on the storage device, and the request includes a read request in which data is transferred from the storage device to the host via the direct memory access transfer.

[0080] Example 19. A method according to any of the embodiments described herein, wherein the request includes a write request, and the data is stored on the storage device via the write request, thereby transferring the data from the host to the storage device via the direct memory access transfer.

[0081] Example 20. A method according to any of the embodiments described herein, wherein the access includes enabling a storage device emulator to coordinate access to the data by using two direct memory access transfers of the data.

[0082] Example 21. A method according to any of the embodiments described herein, wherein the functionality residing on the storage device emulator is used for:

[0083] a. Provide the simulated storage device to the host;

[0084] b. Obtain requests published on the emulated device from the host;

[0085] c. Parse the request; and

[0086] d. Prepare a request formatted for a network storage protocol, including a special memory key (MKEY) pointing to host memory rather than DPU memory, thereby facilitating simple copying of data from one buffer to another on the host within a Remote Direct Memory Access (RDMA) protocol, without requiring functionality residing on the structured target service to be aware of the simple copy.

[0087] Example 22. A method according to any of the embodiments described herein, wherein the functionality residing on the structural target service is used for:

[0088] a. Obtain a request for formatting a network storage protocol from the function residing on the storage device emulator;

[0089] b. Implement DMA transfer of RDMA data to a temporary buffer in host memory, which actually results in a simple copy of the data from one buffer to another in the host without requiring any functionality residing on the target service of the structure to be aware of the simple copy;

[0090] c. Generate a request formatted for the local storage protocol; and

[0091] d. The request, formatted for the local storage protocol and generated by the structure target service, is published to a locally attached hardware driver formatted for the local storage protocol, which causes the locally attached storage device to initiate DMA between the host buffer and internal flash memory.

[0092] Example 23. A method according to any of the embodiments described herein, wherein the computerized apparatus includes a DPU, the DPU including a storage device emulator.

[0093] Example 24. A method according to any of the embodiments described herein, wherein the hardware storage device includes NVMe, and the storage device emulator of the computerized apparatus includes an NVMe driver emulator that emulates an NVMe driver on a PCIe bus.

[0094] Example 25. A method according to any of the embodiments described herein, wherein the DPU locates at least a portion of the data referenced by a request issued by the host via the emulated storage device, the at least portion of the data being available in or targeted in at least one storage device locally attached to the host but not attached to the DPU.

[0095] Example 26. A method according to any of the embodiments described herein, wherein the DPU uses a structured target service provided by the host to access a separate host-attached storage device, thereby generating a set of DMAs, including at least one and no more than three DMAs. Attached Figure Description

[0096] Figure 1 This is a simplified block diagram illustrating a data center environment suitable for implementing embodiments of the present invention.

[0097] Figure 2 It is a simplified block diagram of a data stream that includes three direct memory access transfers in response to a data request.

[0098] Figure 3 This is a simplified block diagram illustration of a data flow according to any embodiment of the present invention, which includes only a single direct memory access transfer.

[0099] Figure 4 This is a simplified block diagram illustration of a data stream comprising two, rather than three, direct memory access transfers according to any embodiment of the present invention.

[0100] Figure 5a and Figure 5b These are simplified flowchart illustrations of algorithms that can be executed separately by functions residing on the computerized device and functions residing on the structural target service. Together, these methods aim to ensure that fewer than three data transfers occur. The methods relate to write I / O, but with necessary modifications, they are equally applicable when the request is a read request rather than a write request. All or any suitable subset of the operations shown can be performed in any suitable order, such as the order shown.

[0101] Figure 6 This is a simplified block diagram illustrating the operation of unloading the structural target. Detailed Implementation

[0102] For simplicity, this specification refers to NVMe by way of example. However, such references are not intended to be limiting, and more generally, any suitable local or PCI storage protocol may be used.

[0103] For simplicity, this specification uses NVMf, also known as NVMe-oF, as examples. However, such references are not intended to be limiting, and more generally, any suitable network storage protocol can be used.

[0104] For simplicity, this manual uses examples to refer to SNAP. TMHowever, such a reference is not intended to be restrictive, and more generally, any storage device emulator (e.g., a smart NIC service) can be used to emulate an NVMe (or virtio-blk) storage PCIe device for the host by exposing the storage PCI device interface to the host (or enabling the host to access the storage PCI device).

[0105] As used herein, the term "smartNIC" can encompass any network interface card (i.e., a PCIe card) that is inserted into a server or storage enclosure to enable Ethernet connectivity. If based on a DPU (Data Processing Unit), the SmartNIC, in addition to providing connectivity, also performs network traffic processing on the NIC itself, which would otherwise have to be done by the CPU in the case of a basic NIC. It is understood that a DPU-based SmartNIC can be based on an ASIC, FPGA, or System-on-a-Chip (SoC).

[0106] The references to "emulated" or "emulating" storage devices in this document are intended to include PCI device interfaces provided to hosts. Therefore, a (hardware) device initially designed as a network device can also be used as a storage device because it exposes or implements access to a storage interface. It can be understood that if a device exposes or implements access to an interface of type X, then the device itself can be considered a type X device. For example, SmartNIC devices typically have or are expected to have a NIC interface (and similarly, NVMe devices are expected to have an NVMe interface, etc.); however, in practice, due to the programmable nature of SmartNIC devices, other interfaces can be provided, which adds this flexibility or emulation capability to the SmartNIC device hardware.

[0107] In some use cases, SNAP can run on a SmartNIC. TM An NVMe drive residing on the host can be used. Typically, when this happens, the solution is for the host to provision the NVMe drive as network storage, and the SmartNIC then connects to that network storage, just as any other network client might do. However, as... Figure 2 As shown, this solution has the disadvantage of transferring data three times on the PCI interface; note that... Figure 2 The following three passes through PCI are shown:

[0108] 1. From host raw application buffer to SmartNIC buffer

[0109] 2. SmartNIC buffer to host NVMf target temporary buffer

[0110] 3. Transfer the host NVMF target temporary buffer to the local NVMe device.

[0111] Some embodiments in this document attempt to achieve the same functionality with only two such transfers or even just a single PCI transfer, where data is passed directly from the host's raw application buffer to the local NVMe. Both possibilities result in better performance.

[0112] Some embodiments enable a host to access an emulated PCI storage device using fewer than three direct memory access (DMA) transfers by using a computerized means for coordinating the execution of host requests arriving at the emulation interface, wherein the host request relates to data stored or to be stored on at least one local hardware storage device directly attached to the host.

[0113] As used herein, the terms “local,” “locally attached,” and “directly attached” are intended to include situations where a (typically hardware) storage device is available to the host via a peripheral bus (such as a PCIe bus) whose root point is the host. Conversely, if a storage device is available to the host only via a network, the storage device may not (e.g., even if its root is connected to the host via a second PCI bus) be considered local to the host, may not be considered locally attached to the host, and may not be considered directly attached to the host. Furthermore, if a storage device is available to the host only via a PCIe bus whose root is not the host, the storage device may not (e.g., even if its root is connected to the host via a second PCI bus) be considered local to the host, may not be considered locally attached to the host, and may not be considered directly attached to the host.

[0114] It's understandable that the number of Direct Memory Access (DMA) transfers can be identified, for example, as follows: If an additional PCI slot is available, a PCI analyzer can be plugged into the same PCI bus as the DPU, and the NVMe drive associated with the transfer is connected. Then, after a simple single read or write I / O request issued to the simulated storage device, the number of DMA transfers (whether they are 1, 2, or 3) can be clearly seen on the analyzer's output. If the CPU and the NVMe drive are not both on the same PCI, two PCI analyzers can be used, one for the DPU and one for the NVMe device. The NVMe device can be unplugged from and plugged back into the slot using an existing raiser in between, and this raiser can also be connected to the analyzer.

[0115] The term "local" is used in this document to refer to a device (typically a hardware storage device) whose data communication with the host is not via a network, and alternatively, often because the device is typically attached directly to the host using a peripheral bus (such as PCIe).

[0116] Typically, the apparatus includes a device emulator configured to emulate a storage device (such as, but not limited to, an NVMe storage PCIe device) to a host. The host in this embodiment may include an NVMe device having a host NVMe drive.

[0117] Figure 1-4 Embodiments of the present invention are shown below. First, Figure 1 A data center with multiple nodes is illustrated. A data center can include any physical group of computer systems and / or associated telecommunications and / or storage systems. For example, in practice, some data centers provide Amazon Web Services, although from the end-user's perspective, these services are provided over the Internet by virtual machines, making the data center providing the service transparent to the remote end-user consuming the service. Typically, a data center includes backup components and / or infrastructure for power supply, and / or data communication connections and / or environmental controls (such as air conditioning and / or fire suppression) and / or security equipment.

[0118] Data can flow from a host on one node to a local drive on that host node (aka process 1), or to remote network storage (aka process 2), or to a drive on another node (process 3). Figure 2-4 The implementation of process 1 is illustrated, where data flows from the host to the local drive (or other hardware storage device) in the host node. It should be recognized that process 1 is particularly suitable for use cases where all I / O is expected to first reach the emulated device (i.e., the device provided by the DPU storage interface). Subsequently, logic on the DPU determines whether a particular request, or a portion thereof, needs to involve the host local drive.

[0119] exist Figure 2 In the process, the regular data stream responding to the request includes three direct memory access transfers, denoted as DMA numbers 1, 2, and 3. For clarity, in Figure 2-4 In this context, a request for data is represented as a write request; however, alternatively, the request could be a read request. It is understood that the target service may or may not be offloaded to hardware.

[0120] exist Figure 3 In some embodiments, the aforementioned fewer than three direct memory access transfers consist of only a single PCI transfer, which allows data to be transferred directly between the host’s original application buffer and a hardware storage device that is local to the host or directly attached to the host. The device coordinates the transfer by enabling the local storage device to directly access the host data buffer residing in the host, including passing a request to transfer to the computerized device to the local storage device.

[0121] exist Figure 3 and Figure 4In the embodiment, SNAP is operated TM The host NVMe drive of the emulated NVMe device can be connected to Figure 2 The same as in. However, as in Figure 3 and Figure 4 The term "modified" is used to indicate this, as indicated by SNAP. TM The process executed by the target service (typically in response to a data request) is related to all or any subset of the following in all or any of the following ways: Figure 2 The execution processes of the corresponding parts in the text are different:

[0122] 1. SNAP TM The NVMF initiator sends an NVMe request (SQE) instead of an NVMe-oF request (command wrapper). Typically, the NVMe request contains a pointer to local system memory, which is initially submitted by the host to SNAP. TM The simulation creates pointers to NVMe devices; and / or requests NVMe using vendor-specific opcodes instead of standard read / write opcodes. Other alternatives can be implemented if needed (e.g., separate predefined network subsystems).

[0123] 2. NVMe-oF target software typically receives requests based on vendor-specific opcodes (or based on any other alternative, such as a separate predefined network subsystem or another indication within the NVMe request), parses the vendor-specific opcodes, and infers that it is an NVMe request rather than an NVMe-oF request.

[0124] 3. NVMf target software does not need to perform network data movement (because the data already resides in the host memory).

[0125] 4. The NVMe target software replaces the opcode (which may be vendor-specific) with a standard read / write opcode (if it is an instruction method) and submits it to the local NVMe device.

[0126] It should be understood that, Figure 3 and Figure 4 In this embodiment, the physical NVMe device accesses the raw application buffer. This can be achieved through a simple pass-through configuration without using any IOMMU or with an IOMMU, allowing the physical NVMe device and SNAP... TM The simulated NVMe devices are all in the same direct domain. Therefore, the same address can be used for both the physical NVMe device and the SNAP. TMIn a simulated NVMe device, both describe the raw application buffer. Conversely, in a virtualized environment, the raw application buffer is part of the virtual machine (VM) memory space. Therefore, appropriate options can be used to ensure that the physical NVMe device has access to the raw application buffer, such as, but not limited to, any of the following:

[0127] a. Use PASID (Process Address Space ID) technology on the physical NVMe disk side;

[0128] b. Use a physical NVMe disk that supports Virtual Functions (VF) and place such a VF in the same domain as the VM;

[0129] c. Create a new Input / Output Memory Management Unit (IOMMU) domain, which includes the VM memory domain, typically all memory domains for all VMs, and place the physical NVMe devices within this domain. Then, for example, in SNAP... TM In the software, the raw address can be translated into the address of the correct VM memory on the new domain (including all of them).

[0130] d. Obtain the VM memory mapping table of the machine address and translate it into SNAP. TM In the middle. Then, SNAP TM The host address can be translated based on this mapping and used in physical NVMe requests.

[0131] e. Input / output memory management unit that obtains addresses using ATS (Address Translation Service).

[0132] (IOMMU) conversion.

[0133] In order to generate Figure 3 (or Figure 4 The modified target service, with the functionality described herein added to it, can be achieved using a target subsystem, such as by writing NVMExpress-compliant software using NVM Express Consortium literature, and can run on a host system capable of data communication with the DPU. The DPU can be plugged into the server in any suitable manner and can connect to the host system in any suitable manner.

[0134] The NVM Express standard can be used to standardize non-standard NVMe-oF requests and allow such requests to include pointers to host memory instead of pointers to remote node memory. In both the NVMe and NVMe-oF standards, a request is defined as including either of two options: an expression pointer, and if NVMe-oF RDMA is used, the addressing scheme is remote memory (e.g., in the NVMe standard, this could be referred to as "keyed SGL"). However, removing this requirement would allow the NVMe-oF RDMA protocol to pass standard local host addresses. Because local host addresses are memory of the storage system, the standard can be enhanced with appropriate security features and negotiation that allow certain addresses to be used and prohibit the use of certain others. If the target function is part of hardware offloading, an internally programmable CPU can be programmed to perform the target service function described herein.

[0135] The target service can run, for example, on an x86 host.

[0136] In order to generate Figure 3 (or Figure 4 Modified SNAP TM (or more generally, computerized devices), for employing The developed tools and SDKs can be used to provide the functionality described in this article that resides on computerized devices.

[0137] According to some embodiments, host access to emulated PCI storage devices may generate multiple direct memory access transfers, wherein the access includes enabling computerized devices to coordinate access to data.

[0138] It is understandable that a single request can span multiple host-connected drives. Alternatively or additionally, it is possible to use host drives to serve fewer than all published requests (e.g., requests published by hosts via emulated storage devices) because a portion of the requests can still be served from the network.

[0139] Therefore, according to an embodiment, the function residing on the computerized device can be used to perform at least one of the following:

[0140] a. Provide the simulated storage device to the host;

[0141] b. Obtain requests published on the emulated devices from the host;

[0142] c. Parse the request;

[0143] d. Determine that one or more storage devices attached to the host are the source or destination of at least a portion of the data referenced in the request, and in response, determine that one or more storage devices attached to the host should be involved in the request; and

[0144] e. Prepare at least one request having a local memory address pointing to a raw buffer located in the host's memory, for sending to a local hardware storage device attached to the host.

[0145] It should be recognized that all or any subset of the above operations can be performed, such as, but not limited to, all operations ae or operations ac and e without d.

[0146] According to some embodiments, a request with a local memory address is formatted for a network storage protocol, wherein a function residing on a structured target service for data communication with a computerized device can be used to:

[0147] a. Identify non-standard NVMF requests by recognizing new opcodes or instructions within existing opcodes;

[0148] b. Accordingly, a regular NVMe request formatted for the local storage protocol (e.g., with a network address) is generated using the pointer to the original buffer received in the non-standard NVMe request; and

[0149] c. Post a regular NVMe request to a locally attached storage device managed by a local storage protocol, which will cause the locally attached storage device to initiate at least one DMA transfer of data between the host buffer and the internal flash memory.

[0150] According to some embodiments, the network storage protocol includes block storage network storage protocols, such as Remote Direct Memory Access (RDMA) network block storage protocols from, but not limited to, the following group: NVMe-oF, iSER, SRP. Local storage protocols may include, but are not limited to, PCI storage protocols from, but not limited to, the following group: NVMe, Virtio-blk, Virtio-scsi, SCSI, SATA, SAS, IDE.

[0151] exist Figure 4 In one embodiment, access includes enabling the storage device emulator to coordinate access to the data by using two direct memory access transfers of the data.

[0152] According to some embodiments, the functionality residing on the storage device emulator can be used for:

[0153] a. Provide the simulated storage device to the host;

[0154] b. Obtain requests published on the emulated devices from the host;

[0155] c. Parse the request; and

[0156] d. Prepare a request for formatting against the network storage protocol, including a memory key (MKEY) pointing to the host memory, for example, instead of pointing to the DPU memory as a regular memory key, thereby facilitating a simple copy of data from one buffer to another on the host within the RDMA protocol, without requiring functions residing on the structured target service to be aware of this simple copy.

[0157] It is understandable that any suitable method can be used to generate and use a memory key (MKEY) pointing to the host memory. For example, for a DPU... The appropriate firmware call generates an MKEY that spans the memory of another system, not the caller's memory (e.g., during memory execution). Figures 5a to 5b (The application's memory runs on the internal processing unit of the DPU of the method and / or other methods described herein). This MKEY conforms to the MKEY requirements specified by RDMA and can therefore be used just as any regular MKEY would be used.

[0158] Functions residing on the structural target service are typically used for:

[0159] a. Obtain a request for formatting according to the network storage protocol from a function residing on the storage device emulator;

[0160] b. Implement DMA transfer of RDMA data to a temporary buffer in host memory, which actually results in a simple copy of data from one buffer to another in the host without requiring any functions residing on the structure target service to be aware of this simple copy;

[0161] c. Generate a request formatted for the local storage protocol; and

[0162] d. The request generated by the structure target service for local storage protocol formatting is published to the locally attached hardware driver formatted for local storage protocol, which will cause the locally attached storage device to initiate DMA between the host buffer and the internal flash memory.

[0163] Typically, a computerized device includes a storage device emulator (DPU). The DPU may include... DPU, and typically, local storage devices include NVMe, and the storage device emulator for computerized devices includes an NVMe driver emulator that emulates the NVMe driver on the PCIe bus. For example, an NVMe driver emulator may include Mellanox NVMeSNAP. TM equipment.

[0164] As used in this article, software-defined network acceleration processing (also known as SNAP) TMThis can include any subsystem or technology that implements hardware-accelerated virtualization of NVMe storage, making the networked storage appear as a local NVMe SSD, for example, by emulating an NVMe drive or other storage device on a PCIe bus. The host OS / hypervisor typically uses its regular NVMe driver, unaware that the communication is not driven by a physical driver but by NVMe SNAP. TM Termination. Any logic can be achieved via NVMe SNAP. TM The framework is applied to the data and transmitted over the network to the storage target via Ethernet or unlimited bandwidth protocols.

[0165] NVMe SNAP TM This allows end users to implement their own storage solutions on top of the provided framework. NVMe SNAP TM Expose or implement access to the kernel's interface (e.g., provide access to the kernel-to-interface), such as... SmartNIC The core is used to control the storage solution. NVMe SNAP TM It integrates with the popular Storage Performance Development Kit (SPDK) open-source project, providing customers with the agility to program in a familiar environment. Typically, end users are provided with one or both of two data paths—the first being completely offloaded, utilizing NVMeSNAP... TM Hardware offloading, the NVMe SNAP TM Data traffic is acquired from NVMe PCIe, converted to NVMe-oF (e.g., RoCE (RDMA over Aggregated Ethernet) or Infiniband), and transmitted directly to the network—typically all within the hardware. It should be recognized that this option may lack the ability for software running on the ARM core to "touch" the data or change the storage protocol. A second data path is available that allows an SPDK running on the ARM core to terminate traffic from NVMe PCIe, enabling any client logic to be implemented on the NVMe PCIe, and then transmit the data to the network. This path, using the ARM core, allows for the flexibility of implementing any type of storage solution online. In both data path options described above, the control plane typically always runs on the ARM core, coordinating traffic to its destination.

[0166] End users can Develop custom virtualization solutions on top of SmartNICs, including NVMe SNAP. TM Framework utilization The SoC uses an ARM core to utilize its built-in hardware acceleration engine.

[0167] NVMe SNAPTM Characterized by all or any subset of the following:

[0168] • Achieve hardware internal storage virtualization

[0169] Programmable NVMe SNAP TM The framework enables integration into any storage solution that can be based on any network protocol.

[0170] • Enables optimization of storage resources used for CAPEX and OPEX savings

[0171] • Free up compute node CPU resources by unloading both network data paths and storage data paths.

[0172] · NVMe SNAP TM With intelligent NICs (such as A two-in-one solution combining SmartNIC and other technologies:

[0173] Dual-port 25 / 100 Gb / s network adapter card

[0174] SoC: 16 ARM A72 cores

[0175] Hardware Unloading Accelerator

[0176] DDR4 memory

[0177] PCIe 3.0 / 4.0 interfaces

[0178] FHHL shape factor

[0179] • Use cases, including all or any subset of the following:

[0180] Bare metal cloud storage virtualization

[0181] Support scaling design with zero software impact

[0182] Introducing NVMe-oF to enterprises that do not have OS type / version dependencies

[0183] --Data centers achieve better server utilization, allowing more virtual machines and more tenants on the same hardware, while reducing TCO, power consumption and cabling complexity.

[0184] NVMe SNAP TM It can be based on Mellanox, for example. This technology combines hardware-accelerated storage virtualization with... SmartNIC combines advanced networking and programmability. It features NVMe SNAP. TM of The SmartNIC functions as an intelligent network adapter for both storage and network virtualization, thereby providing in-hardware storage virtualization to improve both storage and networking infrastructure, as Mellanox does. NVMe SNAP TM This enables hardware-in-memory virtualization while leveraging the flexibility of ARM programmability through smart adapters. Customers can also utilize it in parallel. Infrastructure enables network virtualization offloading, such as running vSwitch controllers on ARM cores while offloading data paths to the SoC. Technology that enables virtualization to maximize scalability and efficiency.

[0185] It should be recognized that the DPU can locate at least a portion of the data referenced by a request issued by the host via an emulated storage device, such as data available in or for at least one storage device locally attached to the host rather than the DPU.

[0186] It is understood that the requested data may not involve any host storage devices at all. Alternatively, at least a portion of the requested data (all or only a portion of the requested data) may involve one or more host storage devices. For a particular portion (or portions) of the requested data, a single host device or multiple host devices may be involved.

[0187] According to some embodiments, the DPU uses a structured target service provided by the host to access a separate host-attached storage device, thereby creating a set of DMAs including 1-3 DMAs.

[0188] It is understood that a request can be either a read request or a write request. In the first instance, typically, data is stored on a storage device, and the request includes a read request in which data is transferred from the storage device to the host via direct memory access transfer. However, alternatively, a given request may include a write request, and data is stored on the storage device via the write request, thereby transferring data from the host to the storage device via direct memory access transfer.

[0189] According to one embodiment, the system operates in conjunction with a computerized device and a structured target service that communicates data with the computerized device, and the system includes functions residing on the computerized device and functions residing on the structured target service, which enable the computerized device to coordinate access to data when operating in combination.

[0190] Example: A computerized device typically includes: a storage device emulator that emulates a storage device on a PCIe bus; and a structured target service, wherein the storage device emulator includes a driver emulator that emulates a driver on the PCIe bus. The driver emulator may include an NVMe driver emulator that emulates an NVMe driver on the PCIe bus. Typically, the NVMe driver emulator employs software-defined network acceleration processing. The NVMe driver emulator may, for example, include Mellanox NVMe SNAP. TM Equipment. Optionally, the structure target service is provided by the host and local hardware storage devices, and is assisted by dedicated structure target hardware offloading.

[0191] Structural target unloading can be done according to Figure 6 This process typically involves only the control path, administration, and exception handling through the target CPU software. Data paths and NVMe commands are usually handled by the network adapter.

[0192] Typically, a target service can be used to provide the storage system to network clients, handle connections arriving from clients, handle management tasks (e.g., the management command set defined in the NVMe and NVMe-oF specifications), and process / serve each IO request using the configured physical drives. In the target offloading feature provided according to certain embodiments, the target service is offloaded to the hardware that subsequently processes / serves IO requests, rather than the host doing so. Typically, the host still runs software to handle connection and management tasks, and on suitable hardware (e.g., ... -5 and The target offload feature is configured in the software; however, once this is complete, the task of processing each request (including servicing requests using the configured physical driver) is entirely handled by the hardware, while the host software remains responsible for abnormal flows, errors, disconnections, etc. The configuration for NVMe over Fabrics (NVMe-oF) target offload is described online, for example at community.melinox.com / s / article / howto-configuration-nvme-over-fabrics--nvme-of--target-offload.

[0193] It is understood that, generally speaking, references to driver emulators in this document are merely illustrative, as more generally, any storage device (not necessarily a drive) can be emulated. Furthermore, references to NVMe in this document are merely illustrative, as more generally, any local storage protocol can be used. Similarly, references to NVMe-oF in this document are merely illustrative, as more generally, any network storage protocol can be used.

[0194] Functions residing on a computerized device (whether provided independently or in combination with other components shown and described herein) are typically used to perform methods. Figure 5a (As shown), the method includes all or any subset of the following operations in an appropriate order, such as the following:

[0195] a. Provide the simulated storage device to the host;

[0196] b. Obtain requests published on the emulated devices from the host;

[0197] c. Parse the request;

[0198] d. Determine that one or more storage devices attached to the host are the source or destination of at least a portion of the data referenced in the request, and in response, determine that one or more storage devices attached to the host should be involved in the request; and

[0199] e. Prepare at least one request having a local memory address pointing to a raw buffer located in the host's memory, for sending to a local hardware storage device attached to the host.

[0200] Functions residing on the structural target service are typically used to perform all or any subset of methods that include the following operations: Figure 5b As shown in the diagram, these operations are appropriately ordered, for example as follows:

[0201] a. Identify requests with local memory addresses by recognizing new opcodes or instructions within existing opcodes;

[0202] b. Therefore, using the pointer to the original buffer received in the request with the local memory address, a regular request formatted for the local storage protocol (with a network address) is generated; and

[0203] c. Post a regular request (with a network address) to the locally attached storage device managed by the local storage protocol, which will prompt the locally attached storage device to initiate at least one DMA transfer of data between the host buffer and the internal flash memory.

[0204] The functions on a computerized device can be implemented using software, hardware, firmware, or any combination thereof. The functions on a structural target unloading device can be implemented using software, hardware, firmware, or any combination thereof.

[0205] It should be understood that Figure 5a and 5b The method is generally simpler than the following method, which generates three or more direct memory access transfers:

[0206] i. The functions on the computerized device, with the help of HW capabilities, provide the host with emulated storage devices (NVMe in this case).

[0207] ii. Functions on the computerized device, with the aid of HW capabilities, obtain requests from the host deployed on the emulated device.

[0208] iii. The computerized device resolves the request.

[0209] iv. With the aid of HW capabilities, the DMA data buffer (DMA#1) is requested by the internal DPU memory.

[0210] v. Prepare a standard NVMe-oF request and send it to the target service.

[0211] Combine all or any subset of the following operations:

[0212] Operation 1011: The structure target service function obtains the NVMe-oF regular request (with network address) prepared in the above operations. Operation 1012: RDMA data (DMA#2) from DPU to the temporary buffer in host memory.

[0213] Operation 1013: The structure target service generates a regular NVMe request.

[0214] Operation 1014: The architecture target service publishes an NVMe request to the locally attached physical NVMe drive, driving DMA data to the internal flash memory (DMA#3).

[0215] As can be seen, this method produces three types of DMA, therefore more than Figure 5a , 5b The method is not very simple.

[0216] Many uses of the embodiments described herein are possible, such as, for example SNAP TM (or other computerized devices) may seek access to host NVMe usage, including caching layers, tiered storage, and hyperconverged usage.

[0217] According to certain non-limiting embodiments, the methods herein utilize all or any subset of the following techniques:

[0218] NVMe

[0219] "NVM Express (NVMe) or Non-Volatile Memory Host Controller Interface Specification (NVMHCIS) is an open logical device interface specification for accessing non-volatile storage media attached via the PCI Express (PCIe) bus."

[0220] NVM stands for Non-Volatile Memory, which is typically NAND flash memory and comes in several physical form factors, including solid-state drives (SSDs), PCI Express (PCIe) add-on cards, M.2 cards, and others.

[0221] Software-defined network acceleration

[0222] NVMe SNAP TM Implement hardware virtualization for NVMe storage. Mellanox NVMe SNAP TM The framework enables customers to easily integrate networked storage solutions into their cloud or enterprise server deployments. NVMe SNAP TM Bringing virtualized storage to the bare-metal cloud and simplifying composable storage. This enables efficient de-aggregation of compute and storage to allow for fully optimized resource utilization, thereby promoting composable storage.

[0223] NVMe SNAP TM Giving customers the freedom to use NVMe SNAP TM Framework (its Mellanox on the chip controller) They implement their own storage technologies and solutions on top of the system (which runs on SNAP). TM By utilizing The embedded hardware memory acceleration engine and integrated programmable ARM core achieve both performance and software transparency. This powerful combination is agile yet completely transparent to the host software, thus allowing SNAP... TM It is integrated into various storage solutions.

[0224] Data processing unit

[0225] Data Processing Units (DPUs) are driving unprecedented innovation in modern data centers, providing a wide range of advanced networking, storage, and security services for complex computing and AI workloads. This is achieved through industry-leading... The combination of network adapter and ARM core array It provides a specially built hardware acceleration engine with full on-chip programmability for data center infrastructure.

[0226] Benefits include all or any subset of the following:

[0227] a. Data storage for scaling workloads; utilizing NVMe over Fabric (NVMe-oF) for direct, encrypted, elastic storage, data integrity, compression, and deduplication. -2DPU provides a high-performance storage network with latency to compete with remote storage for direct-attached storage.

[0228] b. High-performance, high-efficiency networking; -2DPU is a powerful data center service accelerator that delivers up to 200 gigabits per second (Gb / s) Ethernet and unlimited bandwidth line speeds for both traditional applications and modern GPU-accelerated AI workloads, while freeing up host CPU cores; and

[0229] c. Software-defined infrastructure; The DOCA Software Development Kit (SDK) enables developers to easily create high-performance, software-defined, cloud-native, DPU-accelerated services using industry-standard APIs.

[0230] NVMe over Fabric (also known as NVMe-oF or NVMF)

[0231] Devops.com describes NVMe as a protocol that specifies how a CPU moves memory to a storage device via the PCI bus. NVMe communicates through a set of rings (per CPU), where commands can be submitted from any CPU to the underlying NVMe device. NVMe is designed to eliminate the intermediate layer between the CPU and the storage device. An NVMe device consists of a controller, queues, namespaces, namespace IDs, and the actual storage medium with some form of interface. The storage medium can be grouped into parts with IDs called namespaces. In the context of NVMe, namespaces provide a way to enforce access control for disks / consumers. Namespaces are similar to OS partitions, except that partitioning is done in hardware by the controller, not the OS (though OS partitions can still exist on namespaces). Some NVMe namespaces can be hidden from the user (e.g., for security isolation). Controllers connect to ports via queues and to namespaces via their namespace IDs. Controllers can connect to multiple namespaces, and namespaces can be controlled by multiple controllers (and therefore multiple ports). Imagine wiping this NVMe device across multiple computers; you get the next important concept: storage architecture.

[0232] …When you place the network between the PCI bus and the storage device, you use NVMe over Fabric (also known as NVMe-oF or simply NVMF). NVMF enables fast access between the host and storage system over the network. Compared to iSCSI, NVMF has significantly lower access latency, adding only a small latency difference between local and remote storage in practice. NVMF represents a breakthrough in throughput and seek time compared to storage attached to traditional devices.”

[0233] Cache layer

[0234] Cache tiering involves combining erasure coding or slower / lower-cost devices, which are used as a more economical storage tier than the cache tier, into a "backup pool" that provides fast and / or expensive storage devices, such as solid-state drives, configured to operate as a cache tier. For example, Mellanox offers a storage acceleration software product called VSA, a software platform built around iSER technology. VSA is designed to support the use of flash memory or SSDs as a cache tier.

[0235]

[0236] Tiered storage is a known storage technology. For example, Mellanox's... The 40Gb / s unlimited bandwidth adapter provides leading I / O performance for RAID X2-IB (a new unlimited bandwidth tiered storage solution).

[0237] Hyperconvergence, or hyper-aggregation technology, typically involves aggregating all or any subset of compute, storage, networking, and virtualization into an infrastructure that is usually invisible. This technology moves away from proprietary and expensive storage arrays to open up standard compute and storage architectures built around off-the-shelf commodity servers. Organizations can use commodity architectures to achieve hyperconverged solutions that compete with large, expensive storage arrays, while maintaining performance consistent with typical storage platforms. Microsoft's Storage Space Direct (S2D) version in Windows Server 2019 is an example of a hyperconverged solution.

[0238] It should be understood that, if desired, the software components of this invention can be implemented in ROM (Read-Only Memory) form. If desired, the software components can typically be implemented in firmware or hardware using conventional techniques. It should also be understood that the software components can be instantiated, for example, as a computer program product or on a tangible medium. In some cases, it is possible to instantiate the software components as signals that can be interpreted by a suitable computer, although such instantiation may be excluded in certain embodiments of this invention.

[0239] It should be understood that, for clarity, the various features of the invention described in the context of individual embodiments may also be provided in combination in a single embodiment. Conversely, for brevity, the various features of the invention described in the context of individual embodiments may also be provided individually or in any suitable sub-combination.

[0240] Those skilled in the art will understand that the present invention is not limited to what has been specifically shown and described above. Rather, the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A computerized system that operates in conjunction with a computerized device and a structured target service that communicates data with said computerized device, said system comprising: Functions residing on the computerized device; as well as Functions residing on the structural target service When combined, the function enables the computerized device to coordinate access to data through the following steps: Create an Input / Output Memory Management Unit (IOMMU) domain that includes memory domains of multiple virtual machines; as well as At least one raw address requested from the host is translated to an address that matches at least one of the memories of the plurality of virtual machines in the IOMMU domain, wherein the computerized device operates to copy data from one buffer on the host to another within a Remote Direct Memory Access (RDMA) protocol, without requiring any function residing on the structure target service to be aware of the copy.

2. The computerized system of claim 1, wherein the computerized system enables a host to access a emulated PCI storage device by using the computerized means for coordinating the execution of a host request arriving at an interface of the emulation, using fewer than three Direct Memory Access (DMA) transfers, wherein the host request relates to data stored or to be stored on at least one hardware storage device locally attached to the host, wherein the data is transferred between the host's original application buffer and the hardware storage device locally attached to the host.

3. The computerized system of claim 2, wherein the function residing on the computerized device is used to perform at least one of the following: a. Provide the simulated storage device to the host; b. Obtain requests published on the emulated device from the host; c. Parse the request; d. Determine that one or more hardware storage devices attached to the host are the source or destination of at least a portion of the data referenced in the request, and in response, determine that the one or more storage devices attached to the host should be involved in the request; as well as e. Prepare at least one request having a local memory address pointing to the original buffer located in the host's memory, to be sent to the one or more hardware storage devices attached to the host.

4. The computerized system of claim 3, wherein the request having a local memory address is formatted against a network storage protocol, and wherein the function residing on the structure target service is used for: a. Identify the request with a local memory address by recognizing a new opcode or indication within an existing opcode; b. Accordingly, using a pointer to the original buffer received in the request having a local memory address, a regular request formatted for the local storage protocol is generated; as well as c. The regular request is published to the locally attached storage device managed by the local storage protocol, which will cause the locally attached storage device to initiate at least one DMA transfer of data between the host buffer and the internal flash memory.

5. The computerized system according to claim 1, wherein the computerized device comprises: A storage device emulator that simulates storage devices on the PCIe bus; as well as The structural target service. Furthermore, the storage device emulator mentioned above includes a driver emulator that emulates drivers on the PCIe bus.

6. The computerized system of claim 5, wherein the driver emulator includes an NVMe driver emulator that emulates an NVMe driver on a PCIe bus.

7. The computerized system of claim 6, wherein the NVMe driver emulator employs software-defined network acceleration processing.

8. The computerized system of claim 1, wherein the structure target service is provided by a host locally attached to a hardware storage device, and the structure target service further includes a structure target hardware offloading that performs at least some functions on behalf of the structure target service.

9. The computerized system of claim 2, wherein the hardware storage device comprises a physical NVMe device and the emulated PCI storage device are in the same passthrough domain, and wherein the same address is used to describe the host's raw application buffer for both the physical NVMe device and the emulated PCI storage device.

10. The computerized system of claim 9, having a physical NVMe side, wherein the original application buffer is part of a virtual machine (VM) memory space or VM domain, the emulated PCI storage device is located in the VM domain, and wherein the physical NVMe device located in a passthrough host domain rather than the VM domain accesses the original application buffer using Process Address Space ID (PASID) technology on the physical NVMe side.

11. The computerized system of claim 9, wherein the original application buffer is part of a virtual machine memory space or a VM domain, the emulated PCI storage device is located in the VM domain, and wherein the physical NVMe device accesses the original application buffer by creating an Input / Output Memory Management Unit (IOMMU) domain comprising memory domains of multiple virtual machines, deploying the physical NVMe device in the IOMMU domain, and translating at least one original address to an address that matches at least one of the memory domains of the multiple virtual machines.

12. The computerized system of claim 9, wherein the original application buffer is part of a virtual machine memory space or a VM domain, the emulated PCI storage device is located in the VM domain, and wherein the physical NVMe device located in a passthrough host domain rather than in the VM domain accesses the original application buffer by using an Address Translation Service (ATS) to provide input / output memory management unit (IOMMU) translation for at least one address.

13. The computerized system of claim 4, wherein the local storage protocol includes PCI storage protocols from the group consisting of NVMe, Virtio-blk, Virtio-scsi, SCSI, SATA, SAS, and IDE.

14. The computerized system of claim 4, wherein the network storage protocol includes a block storage network storage protocol.

15. The computerized system of claim 14, wherein the block storage network storage protocol includes remote direct memory access (RDMA) network block storage protocols from the group consisting of NVMe-oF, iSER, and SRP.

16. A method for enabling a host to access an emulated PCI storage device, the method comprising: A computerized device is used to coordinate the execution of host requests arriving at the emulated interface, while using fewer than three direct memory access (DMA) transfers, wherein the host request relates to data stored or to be stored on at least one hardware storage device locally attached to the host, and wherein the host request includes vendor-specific opcodes. Create an Input / Output Memory Management Unit (IOMMU) domain that includes memory domains of multiple virtual machines; as well as At least one raw address requested from the host is translated into an address that matches at least one of the memories of the plurality of virtual machines in the IOMMU domain; as well as The new opcode is converted into a different, existing opcode, wherein the new opcode includes vendor-specific opcodes, wherein the access includes enabling the storage device emulator to coordinate access to the data using two direct memory access transfers of the data, and wherein the functionality residing on the storage device emulator is used for: a. Provide the simulated storage device to the host; b. Obtain requests published on the emulated device from the host; c. Parse the request; as well as d. Prepare a request formatted for a network storage protocol, including a special memory key MKEY pointing to host memory instead of DPU memory, thereby facilitating simple copying of data from one buffer to another on the host within the Remote Direct Memory Access (RDMA) protocol, without requiring functionality residing on the structured target service to be aware of the simple copying.

17. The method of claim 16, wherein the host uses a plurality of direct memory access transfers to access the emulated PCI storage device, and wherein the access includes enabling the computerized device to coordinate access to the data.

18. The method of claim 16, wherein the data is stored on the storage device, and the request includes a read request in which data is transferred from the storage device to the host via the direct memory access transfer.

19. The method of claim 16, wherein the request includes a write request, and the data is stored on the storage device via the write request, thereby transferring the data from the host to the storage device via the direct memory access transfer.

20. The method of claim 16, wherein the function residing on the structural target service is used for: a. Obtain a request for formatting a network storage protocol from the function residing on the storage device emulator; b. Implement DMA transfer of RDMA data to a temporary buffer in host memory, which actually results in a simple copy of the data from one buffer to another in the host without requiring any functionality residing on the target service of the structure to be aware of the simple copy; c. Generate a request formatted for the local storage protocol; as well as d. The request, formatted for the local storage protocol and generated by the structure target service, is published to a locally attached hardware driver formatted for the local storage protocol, which causes the locally attached storage device to initiate DMA between the host buffer and internal flash memory.

21. The method of claim 16, wherein the computerized device includes a DPU, the DPU including a storage device emulator.

22. The method of claim 21, wherein the hardware storage device comprises NVMe, and the storage device emulator of the computerized apparatus comprises an NVMe driver emulator that emulates an NVMe driver on a PCIe bus.

23. The method of claim 21, wherein the DPU locates at least a portion of the data referenced by a request issued by the host via the emulated storage device, the at least portion of the data being available in or targeted in at least one storage device locally attached to the host but not attached to the DPU.

24. The method of claim 23, wherein the DPU uses a structured target service provided by the host to access a separate host-attached storage device locally attached to the host but not attached to the DPU, thereby generating a set of DMAs, including at least one and no more than three DMAs.

25. The computerized system of claim 10, wherein the host's original application buffer is part of a domain included in the virtual machine's memory space, the emulated PCI storage device is located in the domain included in the virtual machine's memory space, and wherein the physical NVMe device accesses the original application buffer by means of the computerized device, based on a mapping passed to the computerized device by an assistant driver on the host for guest-to-host memory mapping facing the virtual machine, the physical NVMe device being in a pass-through host domain rather than located in the domain included in the virtual machine's memory space.

26. The method of claim 21, wherein the hardware storage device comprises one of the group consisting of NVMe, Virtio-blk, Virtio-scsi, SCSI, SATA, SAS, and IDE, and the storage device emulator comprises one of the group consisting of NVMe, Virtio-blk, Virtio-scsi, SCSI, SATA, SAS, and IDE driver emulators that emulate storage drivers on a PCIe bus.

Images