Hardware unloading device and method for NVMeoF

Abstract

The invention provides an NVMeoF hardware unloading device and method. The method comprises the steps that a remote host end sends an NVMeoF command, and a hardware unloading engine executes packaging, format checking, address conversion, reverse address mapping, direct access and management data transmission operation on the command; the cache module caches a read message and a write message of the network interface control module RMDA, a command after address conversion and reverse address mapping, and read operation and write operation data; through a network interface control module, a virtual descriptor is extracted and processed, a descriptor extraction task is established, a root node mode request is cached, and a request message and a response message are constructed and verified; and the network interface control module is used for packaging and transforming the ROCE message, submitting a completion message, controlling the transmission rate, configuring transmission parameters and communicating with a remote host end. According to the device and the method, the access delay is reduced while the access performance is improved, and the CPU computing power of the storage board is unloaded.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a hardware unloading device and method for NVMeoF. Background Art

[0002] In today's big data era, massive amounts of data need to be processed in real time, which places more stringent requirements on the bandwidth and throughput of storage systems. In order to meet the new requirements for massive data processing, the performance of solid state drives (SSDs) continues to improve, but the theoretical bandwidth of the traditional SATA (Serial Advanced Technology Attachment) 3.0 interface of 600MB / s has gradually become a performance bottleneck. In comparison, the PCIe (Peripheral Component Interconnection Express) system bus has the characteristics of high bandwidth and low latency, making the combination of PCIe and SSD an inevitable trend. The NVMe (Non-volatile Memory Express) protocol proposed for PCIe SSD has achieved great optimization in capacity, power consumption, scalability and error management, and has become the most important storage method for data-intensive applications such as data centers or supercomputing clouds.

[0003] NVMe is designed for the use and transmission of flash memory on local nodes and for PCIe. Although PCIe is a high-speed interconnect bus that can give full play to the many advantages of the PCIe interface when used locally, as an internal bus, PCIe is limited by the CPU and motherboard, and the number of available PCIe channels is very limited. The CPU also needs to reserve resources for the channels, which further increases the difficulty of expansion and management. Moreover, PCIe is an internal bus that lacks the ability to become a general network, and there is no switch available, which is not suitable for building large-scale storage networks and long-distance data transmission. Under this limitation, NVMe is only applicable to the architecture where computing nodes and storage nodes are integrated, and the CPU directly accesses data to the SSD through the internal system bus. The NVMeoF (NVMe over Fabric) network storage protocol based on the NVMe protocol can be used to communicate with remote NVMe devices between the host and the storage system through various network structures. NVMeoF extends the NVMe protocol on different network protocols, so that in the interconnected NVMe storage system, it can also have the advantages of high performance, low latency and low protocol overhead like the NVMe protocol of the stand-alone system, ensuring that the storage performance of the NVMe device will not be significantly reduced after the NVMeoF protocol is extended. For the remote host, it will occupy less CPU resources, reducing the high requirements for the CPU to handle large amounts of I / O, and the saved CPU resources can be used to process storage-related algorithms; for the back-end storage system, because it is independent of the computing nodes, network nodes and storage devices are scalable and no longer restricted by the PCIe channel, it can be expanded to connect more SSDs, which also reduces the requirements for deployment costs.

[0004] In the current NVMeoF remote data access architecture of data centers, the processing of network data packets is completed by the independent network card at the storage target end. NVMe and NVMeoF protocol processing are completed by the NVMe protocol processing module and the NVMeoF protocol processing module in the operating system running on the CPU respectively. The NVMe storage card, network card and CPU are interconnected via the PCIe bridge chip. In I / O intensive applications, the processing performance of the CPU operating system block I / O layer will become the performance bottleneck for business applications to access peripherals. Especially in asymmetric, many-to-one storage structures, the storage board needs to provide remote data access services to multiple computing boards. The performance of the block I / O layer is likely to become a bottleneck for remote data access; on the other hand, since the operating system will schedule each thread according to the time slice, in long-term and frequent remote data access, the processing thread of the block I / O layer will be scheduled to the non-executing state by the operating system from time to time. At this time, there will be a large number of remote data accesses waiting for system scheduling, which greatly increases the delay of remote access and the uncertainty of processing. The processing performance of the CPU becomes the bottleneck of NVMeoF remote data access. Summary of the invention

[0005] The purpose of the present invention is to provide a hardware unloading device of NVMeoF, which has a simple structure, is safe, effective, reliable and easy to operate, can realize the hardware unloading of RNIC and NVMeoF in the FPGA of the target end, and only implements RDMA driver, NVMeoF driver and NVMe driver on the CPU. Data is directly stored in NVMe SSD via FPGA without sending to DDR of CPU, so that access performance is improved while access delay is reduced, storage board CPU computing power is unloaded, and the performance of I / O layer of remote data access service block for multiple computing boards and the processing performance of CPU become the bottleneck problem of NVMeoF remote data access.

[0006] Based on the above objectives, the technical solution provided by the present invention is as follows: A hardware offloading device of NVMeoF, comprising: a hardware offloading engine, a cache module, a network interface control module and an NVMe storage module; The cache module is connected to the hardware offload engine and the network interface control module respectively; The network interface control module is connected to the remote host end; The hardware offload engine is connected to the NVMe storage module via a PCIe interface; The hardware offload engine is used to sequentially perform encapsulation, format checking, address translation, reverse address mapping, direct access and management data transmission operations on remote host-side NVMeoF commands; The cache module is used to cache the RDMA read message and RDMA write message of the network interface control module, the NVMe command after address conversion and reverse address mapping, the read operation data and the write operation data; The network interface control module is used to extract and process virtual descriptors, establish descriptor extraction tasks, cache root endpoint mode requests, construct request messages and response messages, verify request messages and convert formats, and verify and cache response messages; The network interface control module is also used to encapsulate and construct ROCE messages, submit completion messages, control transmission rates, configure transmission parameters, and perform network communications with remote host terminals.

[0007] Preferably, the hardware offload engine includes a VP queue processing unit, an NVMeoF command analysis mapping unit, an arbitration processing unit, an NVMeoF command processing unit, a data transmission unit and a PCIe EP unit; The VP queue processing unit is used to interact with the network interface control module, receive the NVMeoF command package of the remote host, send the NVMeoF command to the NVMeoF submission queue, and send the corresponding response package to the remote host after the access is completed; The NVMeoF command analysis and mapping unit is used to extract the NVMeoF command from the NVMeoF submission queue, check whether the NVMeoF command conforms to a preset format, and if so, perform address conversion processing to form the NVMe command, and store it in the submission queue of the NVMe storage module; The NVMeoF command analysis and mapping unit is further used to perform reverse address mapping on the submission queue of the NVMe storage module to obtain the NVMe command and store it in the submission queue of the NVMeoF; The arbitration processing unit is used to process the NVMe command and then send it to the NVMeoF queue; The NVMeoF command processing unit is used to process the corresponding NVMeoF queue and interact with the NVMe storage module; The data transmission processing unit is used to manage the data transmission between the hardware offload engine and the cache module.

[0008] Preferably, the cache module includes: a VP cache unit, an NVMe storage module queue cache unit, an NVMeoF queue cache unit and a read-write data cache unit; The VP cache unit is used to cache the RDMA read message and the RDMA write message in the network interface control module; The NVMe storage module queue cache unit is used to cache the NVMe commands after address conversion by the NVMeoF command analysis mapping unit; The NVMeoF queue cache unit is used to cache the NVMe commands after the NVMeoF command analysis mapping unit undergoes reverse address mapping; The read-write data cache unit is used to cache the read operation data and the write operation data generated when the RDMA in the network interface control module performs the read operation and the write operation.

[0009] Preferably, the network interface control module comprises: a descriptor communication processing unit, a reliable transmission control unit, a message construction unit, a message processing unit and an address conversion unit; The descriptor communication processing module is used to extract the request descriptor and the response descriptor from the VP queue processing unit, and to establish a descriptor extraction task based on the retransmission descriptor trigger after scheduling, caching, parsing and checking in sequence; The reliable transmission control unit is used to cache the root endpoint mode request before the root endpoint mode confirmation is completed; The message construction unit is used to construct a request message and / or a response message according to the request descriptor and / or the response descriptor; The message processing unit is used to verify the request message and filter the request message; The message processing unit is further used to check the response message, determine whether it needs to be retransmitted, and transmit the response message that needs to be retransmitted to the read-write data cache unit; The address conversion unit is used to convert the NVMeoF reaction cabin component into the RDMA VP format.

[0010] Preferably, the network interface control module further comprises: a message construction and parsing unit, a CQE processing unit, a congestion control unit, a configuration processing unit and a network interface; The message construction and parsing unit is used to perform a legality check on the RDMA-based Ethernet input message and to encapsulate and construct the RDMA-based Ethernet output message; The CQE processing unit is used to submit the completion message of the inspection or packaging and construction; The congestion control unit is used to control the transmission rate of the network communication of the remote host end; The configuration processing unit is used to set and store the transmission parameter configuration between the CPU of the remote host and the network interface control module; The network interface is used for network communication with the remote host.

[0011] A method for hardware unloading of NVMeoF is implemented based on the hardware unloading device of NVMeoF described in any one of the above items. The method for hardware unloading of NVMeoF includes a write operation of NVMeoF based on RDMA, specifically the following steps: The remote host encapsulates the NVMeoF command into an RDMA message and sends it to the network interface control module; The network interface control module completes the unloading of the RDMA protocol and strips the NVMeoF command encapsulation from the RDMA message; Writing the NVMeoF command encapsulation into a submission queue of a cache module; The hardware offload engine takes out the NVMeoF command encapsulation in the submission queue of the cache module, and completes the protocol offload work of command parsing and address conversion.

[0012] Preferably, after completing the protocol unloading work of command parsing and address conversion, the following steps are also included: The hardware offload engine allocates data space of the read and write data cache unit in the cache module, and writes the RDMA read message into the VP queue processing unit; The remote host reads the data from the memory according to the RDMA read request initiated by the network interface control module, writes the read data into the data space of the read-write data cache unit allocated in the cache module, and notifies the hardware offload engine; The NVMe storage module retrieves the data to be written from the data space of the read-write data cache unit allocated in the cache module according to the doorbell signal sent by the hardware offload engine; After the data transmission is completed, the NVMe storage module writes the completion command into the NVMeoF queue cache unit in the cache module, which is mapped and converted by the hardware offload engine, encapsulated as an NVMeoF reaction cabin component, and written into the NVMeoF completion queue in the cache module; The network interface control module converts the NVMeoF reaction cabin component in the NVMeoF completion queue into an RDMAVP format; The network interface control module sends the completion information in the VP queue processing unit to the remote host end.

[0013] Preferably, the NVMeoF hardware unloading method includes a read operation of NVMeoF based on RDMA, specifically the following steps: The remote host encapsulates the NVMeoF command into an RDMA message and sends it to the network interface control module; The network interface control module completes the unloading of the RDMA protocol and strips the NVMeoF command encapsulation from the RDMA message; Writing the NVMeoF command encapsulation into a submission queue of a cache module; The hardware offload engine takes out the NVMeoF command encapsulation in the submission queue of the cache module and completes the protocol offload work of command parsing and address conversion; The NVMe storage module takes out the NVMe command from the submission queue of the NVMe storage module of the cache module according to the doorbell signal sent by the hardware offload engine, and writes the data into the read-write data cache unit in the cache module after execution; After the data transmission is completed, the NVMe storage module writes the completion command into the NVMeoF queue cache unit in the cache module, which is mapped and converted by the hardware offload engine, encapsulated as an NVMeoF reaction cabin component, and written into the NVMeoF completion queue in the cache module; The network interface control module initiates an RDMA write request to write the data in the read-write data cache unit in the cache module into the remote host memory; The network interface control module converts the NVMeoF reaction cabin component into an RDMA VP format; The network interface control module sends the completion information in the VP queue processing unit to the remote host end.

[0014] The NVMeoF hardware unloading device provided by the present invention is provided with a hardware unloading engine, a cache module, a network interface control module and an NVMe storage module. The remote host sends an NVMeoF command, and the hardware unloading engine performs encapsulation, format checking, address conversion, reverse address mapping, direct access and management of data transmission operations on the command; the cache module caches the NVMe commands, read operations and write operations data after the network interface control module RMDA read messages and write messages, address conversion and reverse address mapping; the network interface control module extracts and processes virtual descriptors, establishes descriptor extraction tasks, caches root node mode requests, and constructs and verifies request messages and response messages; the network interface control module encapsulates and transforms ROCE messages, submits completion messages, controls transmission rate, configures transmission parameters and communicates with the remote host.

[0015] Compared with the prior art, the present invention implements the hardware offload of RNIC and NVMeoF in the FPGA of the target end, and only implements RDMA driver, NVMeoF driver and NVMe driver on the CPU. Data is directly stored in NVMe SSD via FPGA without being sent to the DDR of CPU, thereby improving access performance while reducing access latency, offloading the CPU computing power of the storage board, and solving the problem that the performance of the I / O layer of the remote data access service block for multiple computing boards and the processing performance of the CPU become the bottleneck problem of NVMeoF remote data access.

[0016] The present invention also provides a hardware unloading method of NVMeoF, which belongs to the same inventive concept as the system, solves the same technical problem, and should have the same beneficial effects, so it will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of the structure of a hardware offloading device of NVMeoF provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a hardware offload engine provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a cache module provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a network interface control module provided by an embodiment of the present invention; Figure 5 A flowchart of an RDMA-based NVMeoF write operation in a NVMeoF hardware offload method provided in an embodiment of the present invention; Figure 6 The flowchart after step S4 provided in the embodiment of the present invention; Figure 7 A flowchart of an RDMA-based NVMeoF read operation in an NVMeoF hardware offload method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The embodiments of the present invention are written in a progressive manner.

[0021] The embodiment of the present invention provides a NVMeoF hardware unloading device and method, which mainly solves the technical problem in the prior art that the performance of the I / O layer of the remote data access service block for multiple computing boards and the processing performance of the CPU become the bottleneck of NVMeoF remote data access.

[0022] like Figure 1 As shown, a hardware offloading device of NVMeoF includes: a hardware offloading engine, a cache module, a network interface control module and an NVMe storage module; The cache module is connected to the hardware offload engine and the network interface control module respectively; The network interface control module is connected to the remote host end; The hardware offload engine is connected to the NVMe storage module through the PCIe interface; The hardware offload engine is used to perform encapsulation, format checking, address translation, reverse address mapping, pass-through access and management data transfer operations on the remote host-side NVMeoF commands in sequence; A cache module is used to cache RDMA read messages and RDMA write messages of the network interface control module, NVMe commands after address conversion and reverse address mapping, read operation data, and write operation data; The network interface control module is used to extract and process virtual descriptors, establish descriptor extraction tasks, cache root endpoint mode requests, construct request messages and response messages, verify request messages and convert formats, and verify and cache response messages; The network interface control module is also used to encapsulate and construct ROCE messages, submit completion messages, control transmission rates, configure transmission parameters, and communicate with remote hosts over the network.

[0023] In actual application, the hardware offload device of NVMeoF is equipped with NVMeoF hardware offload engine, cache module Buffer, network interface control module RNIC and NVMe storage module SSD; among them, RNIC is connected to the remote host through the network interface, and the NVMeoF hardware offload engine is connected to the internal PCIe interface through PCIe EP to transmit data with the NVMe SSD storage module.

[0024] It should be noted that NVMe-oF (NVMe over Fabrics) is a non-volatile memory based on architecture. NVMe-oF supports data transmission over the network between the host and solid-state storage devices or systems. NVMe-oF inherits all the advantages of NVMe, including lightweight and efficient command sets, multi-core awareness, and protocol parallelism. NVMe-oF is truly network-independent because it supports all common fabrics, including Fibre Channel, InfiniBand, and Ethernet; RDMA bypasses the kernel and directly accesses the RDMA enabled NIC from user space through a dedicated RDMA network card RNIC (RDMA-aware Network Interface Controller). RDMA provides a dedicated Verbs Interface instead of the traditional TCP / IP Socket Interface. To use RDMA, you must first establish a data path from RDMA to the application memory. These data paths can be established through the RDMA-specific Verbs Interace interface. Once the data path is established, you can directly access the user space buffer.

[0025] like Figure 2 As shown, preferably, the hardware offload engine includes a VP queue processing unit, an NVMeoF command analysis mapping unit, an arbitration processing unit, an NVMeoF command processing unit, a data transmission unit and a PCIe EP unit; The VP queue processing unit is used to interact with the network interface control module, receive the NVMeoF command encapsulation from the remote host, send the NVMeoF command to the NVMeoF submission queue, and after the access is completed, send the corresponding response encapsulation to the remote host; The NVMeoF command analysis and mapping unit is used to extract the NVMeoF command from the NVMeoF submission queue, check whether the NVMeoF command conforms to a preset format, and if so, perform address conversion processing to form an NVMe command, and store it in the submission queue of the NVMe storage module; The NVMeoF command analysis and mapping unit is also used to perform reverse address mapping on the submission queue of the NVMe storage module, obtain the NVMe command, and store it in the submission queue of the NVMeoF; The arbitration processing unit is used to process the NVMe command and then send it to the NVMeoF queue; NVMeoF command processing unit, used to process the corresponding NVMeoF queue and interact with the NVMe storage module; The data transmission processing unit is used to manage the data transmission between the hardware offload engine and the cache module.

[0026] In actual use, the hardware offload engine is provided with a virtual port VP queue processing unit, an NVMeoF command analysis and mapping unit, an arbitration processing unit and a data transmission processing unit; the VP queue processing unit is used to interact with the network interface control module RNIC, receive NVMeoF command encapsulation from the remote host, submit the NVMeoF command to the NVMeoF submission queue SQ, and after the data access is completed, send the corresponding response encapsulation to the remote host; the NVMeoF command analysis and mapping unit takes out the NVMeoF command from the NVMeoF submission queue SQ, checks whether the command meets the preset format requirements, and if not, Feedback error; if it is in compliance, the NVMeoF command analysis and mapping unit performs address conversion processing on the correct NVMeoF command to form an NVMe command; the NVMe command is arbitrated by the arbitration processing component and stored in the NVMeoF submission queue SQ; the NVMeoF command analysis and mapping unit performs reverse address mapping on the submission queue of the NVMe storage module to obtain the NVMe command and store it in the SQ / CQ of NVMeoF; the NVMe command processing unit is responsible for interacting with the NVMe storage module SSD controller, processing the corresponding NVMe SQ / CQ, and completing P2P direct access to the NVMe storage module SSD. The data transmission processing component is responsible for managing data transmission between the NVMe hardware acceleration engine and the cache module Buffer.

[0027] like Figure 3 As shown, preferably, the cache module includes: a VP cache unit, an NVMe storage module queue cache unit, an NVMeoF queue cache unit and a read-write data cache unit; A VP cache unit, used for caching RDMA read messages and RDMA write messages in the network interface control module; An NVMe storage module queue cache unit, used for caching NVMe commands after address conversion by the NVMeoF command analysis mapping unit; The queue cache unit of NVMeoF is used to cache the NVMe commands after the reverse address mapping by the NVMeoF command analysis mapping unit; The read-write data cache unit is used to cache the read operation data and the write operation data generated when the RDMA in the network interface control module performs the read operation and the write operation.

[0028] During actual application, the cache module Buffer is provided with a VP cache unit VP Buffer, an NVMe storage module queue cache unit SSD SQ / CQ, an NVMeoF queue cache unit NVMeoF SQ / CQ Buffer and a read-write data cache unit; the VP Buffer is used to cache the RMDA read message and the RMDA write message in the RNIC module; the NVMeoF SQ / CQ Buffer is used to cache the NVMe commands after reverse address mapping by the NVMeoF command analysis mapping unit; the NVMe storage module queue cache unit SSD SQ / CQ is used to cache the NVMe commands after address conversion by the NVMeoF command analysis mapping unit; the read-write data cache unit specifically includes a read data sub-cache unit and a write data sub-cache unit, the read data cache unit is used to cache the read operation data generated when performing a read operation, and the write data sub-cache unit is used to cache the write operation data generated when performing a write operation.

[0029] like Figure 4 As shown, preferably, the network interface control module includes: a descriptor communication processing unit, a reliable transmission control unit, a message construction unit, a message processing unit and an address conversion unit; The descriptor communication processing module is used to extract the request descriptor and the response descriptor from the VP queue processing unit, and to schedule, cache, parse and verify them in sequence, and then establish a descriptor extraction task triggered by the retransmission descriptor; A reliable transport control unit, configured to cache a root endpoint mode request before the root endpoint mode confirmation is completed; A message construction unit, used to construct a request message and / or a response message according to the request descriptor and / or the response descriptor; A message processing unit, used to verify and filter request messages; The message processing unit is further used to check the response message, determine whether it needs to be retransmitted, and transmit the response message that needs to be retransmitted to the read-write data cache unit; The address translation unit is used to convert the NVMeoF response module into the RDMA VP format.

[0030] In actual application, the network interface control module RNIC is provided with a descriptor communication processing unit, a reliable transmission control unit, a message construction unit, a message processing unit and an address conversion unit. The descriptor communication processing unit extracts the corresponding request descriptor and response descriptor according to the VP buffer unit VP Buffer, and schedules and caches them in turn, performs preliminary parsing and verification, and after completion, it is recommended to trigger the descriptor extraction task based on the retransmission descriptor; the root endpoint mode request is cached by the reliable transmission control unit before the root endpoint RC mode confirmation is completed; the message construction unit constructs a request message or response message according to the request descriptor / response descriptor; the request message is verified by the message processing unit to filter out duplicate messages or invalid messages and leave normal messages; the NVMeoF reaction cabin component of the normal request message is converted into the RDMA VP format by the address conversion unit; the response message is checked by the message processing unit to determine whether it needs to be retransmitted, and the response message that needs to be retransmitted is transmitted to the read-write data cache unit.

[0031] like Figure 4 As shown, preferably, the network interface control module further includes: a message construction and parsing unit, a CQE processing unit, a congestion control unit, a configuration processing unit and a network interface; A message construction and parsing unit, used to perform a validity check on an RDMA-based Ethernet input message, and to encapsulate and construct an RDMA-based Ethernet output message; CQE processing unit, used to submit the completion message of the inspection or packaging and construction; A congestion control unit is used to control the transmission rate of network communication at the remote host end; A configuration processing unit, used to set and store the transmission parameter configuration between the CPU of the remote host and the network interface control module; Network interface, used for network communication with the remote host.

[0032] In actual use, the network interface control module RNIC is also provided with a message construction and parsing unit, a CQE processing unit, a congestion control unit, a configuration processing unit and a network interface; the received ROCE message is checked for legitimacy through the message construction and parsing unit, and the sent ROCE message is encapsulated and constructed; the completion information is submitted through the CQE processing unit; the transmission rate of the network communication of the remote host end is controlled through the congestion control component; the transmission parameter configuration between the CPU of the remote host end and the network interface control module is set and stored through the configuration processing unit; and network communication is carried out with the remote host end through the network interface.

[0033] It should be noted that ROCE and InfiniBand are both network protocol stacks defined by the InfiniBand Trade Association (IBTA). Infiniband is a high-performance network designed for RDMA, which ensures the reliability of data transmission from the hardware level. In order to further leverage the advantages of RDMA, IBTA defined ROCE in 2010. ROCE is a fusion of Infiniband and Ethernet technology. While maintaining the core advantages of Infiniband, it achieves compatibility with existing Ethernet infrastructure. Specifically, ROCE is different from Infiniband in the link layer* and network layer, but in terms of the transport layer and RDMA protocol, ROCE inherits the essence of Infiniband.

[0034] CQE (Capacity Quality Estimator): In some network architectures, especially for more fine-grained control of network resource allocation and performance optimization, specialized CQE algorithms or tools may be used. These tools can estimate the capacity and quality of the network based on multiple parameters (such as SINR, interference level, user density, etc.).

[0035] like Figure 5 As shown, a hardware unloading method of NVMeoF is implemented based on any of the above-mentioned hardware unloading devices of NVMeoF. The hardware unloading method of NVMeoF includes a write operation of NVMeoF based on RDMA, which is specifically the following steps: S1. The remote host encapsulates the NVMeoF command into an RDMA message and sends it to the network interface control module; S2. The network interface control module completes the unloading of the RDMA protocol and strips the NVMeoF command encapsulation from the RDMA message; S3. Encapsulate the NVMeoF command and write it into the submission queue of the cache module; S4. The hardware offload engine takes out the NVMeoF command encapsulation in the submission queue of the cache module and completes the protocol offload work of command parsing and address conversion.

[0036] In step S1 to step S4, the hardware unloading method of NVMeoF includes a write operation of NVMeoF based on RDMA, specifically: the remote host encapsulates the command capsule of NVMeoF into an RDMA message, and sends it to the remote RNIC in the RDMA Send mode; the network interface control module completes the unloading of the RDMA protocol, strips the command capsule of NVMeoF from the RDMA message, and writes the stripped NVMeoF command into the submission queue of the cache module; the NVMeoF command capsule in the submission queue of the cache module is taken out by the hardware unloading engine, and the protocol unloading work of command parsing and address conversion is completed.

[0037] like Figure 6 As shown, preferably, after step S4, the following steps are also included: A1. The hardware offload engine allocates data space of the read and write data cache unit in the cache module and writes the RDMA read message to the VP queue processing unit; A2. The remote host reads the receipt from the memory according to the RDMA read request initiated by the network interface control module, writes the read data into the data space of the read-write data cache unit allocated in the cache module, and notifies the hardware offload engine; A3. The NVMe storage module retrieves the data to be written from the data space of the read-write data cache unit allocated in the cache module according to the doorbell signal sent by the hardware offload engine; A4. After the data transmission is completed, the NVMe storage module writes the completion command to the NVMeoF queue cache unit in the cache module, which is mapped and converted by the hardware offload engine, encapsulated as the NVMeoF reaction cabin component, and written to the NVMeoF completion queue in the cache module; A5. The network interface control module converts the NVMeoF response module in the NVMeoF completion queue into RDMAVP format; A6. The network interface control module sends the completion information in the VP queue processing unit to the remote host.

[0038] In actual use, after completing the protocol offloading work of command parsing and address conversion, the hardware offloading engine allocates the write data sub-cache unit RX Buffer data space in the read-write data cache unit in the cache module Buffer, and writes the RDMA Read message to the RDMA VP; the RNIC initiates RDMA Read to the Host to read data from the Host memory; the Host returns the data, and RDMA writes the received read data to the RX Buffer data space allocated by the Buffer, and notifies the NVMeoF hardware offloading engine; the NVMeoF hardware offloading engine sends a Doorbell to the NVMe SSD to notify the device that there is a command waiting to be processed; the NVMe SSD retrieves the data to be written from the RX Buffer in the Buffer; after the data transmission is completed, the NVMeSSD writes the completion command to the SSD CQ of the Buffer, which is mapped and converted by the NVMeoF hardware offloading engine and encapsulated as the NVMeoF response capsule and written to the NVMeoF CQ of the Buffer; the RNIC converts the response capsule of the NVMeoF CQ into the format of the RDMA VP; the RNIC uses RDMA Send to send the RDMA VP The completion information in is sent to the remote host Host, indicating that the data writing process is completed.

[0039] like Figure 7 As shown, preferably, the hardware offloading method of NVMeoF includes a read operation of NVMeoF based on RDMA, specifically the following steps: B1. The remote host encapsulates the NVMeoF command into an RDMA message and sends it to the network interface control module; B2. The network interface control module completes the unloading of the RDMA protocol and strips the NVMeoF command encapsulation from the RDMA message; B3. Encapsulate the NVMeoF command and write it into the submission queue of the cache module; B4. The hardware offload engine removes the NVMeoF command encapsulation from the submission queue of the cache module and completes the protocol offload work of command parsing and address translation; B5. The NVMe storage module takes out the NVMe command from the submission queue of the NVMe storage module of the cache module according to the doorbell signal sent by the hardware offload engine, and after execution, writes the data into the read-write data cache unit in the cache module; B6. After the data transmission is completed, the NVMe storage module writes the completion command to the NVMeoF queue cache unit in the cache module, which is mapped and converted by the hardware offload engine and encapsulated as the NVMeoF reaction cabin component and written to the NVMeoF completion queue in the cache module; B7. The network interface control module initiates an RDMA write request to write the data in the read / write data cache unit in the cache module to the remote host memory; B8. The network interface control module converts the NVMeoF reaction cabin component into the RDMA VP format; B9. The network interface control module sends the completion information in the VP queue processing unit to the remote host.

[0040] In actual application, the hardware offload method of NVMeoF includes the read operation of NVMeoF based on RDMA, specifically: the remote host end encapsulates the command capsule of NVMeoF into an RDMA message and sends it to the remote RNIC in the form of RDMA Send; the RNIC completes the unloading of the RDMA protocol and strips the command capsule from the RDMA message; the command capsule is written to the submission queue NVMeoF SQ in the storage buffer Buffer; the NVMeoF hardware offload engine takes out the command capsule from the submission queue NVMeoF SQ of the Buffer and completes the protocol offload work such as command parsing and address conversion; the NVMeoF hardware offload engine sends a doorbell signal Doorbell to the NVMe SSD to notify the device that there is a command waiting to be processed; the NVMe SSD takes out the NVMe command from the SSD SQ of the Buffer, and after execution, writes the data into the read data sub-cache unit TX Buffer in the read-write data cache unit of the Buffer; after the data transmission is completed, the NVMe SSD will complete the command writing to the SSD CQ of the Buffer; the NVMeoF hardware offload engine will write the SSD After the command mapping conversion is completed in CQ, it is encapsulated into NVMeoF response capsule and written into Buffer's NVMeoF CQ; RNIC initiates RDMA Write to write the data in Buffer's TX Buffer into Host memory; RNIC converts NVMeoF response capsule into RDMA VP format; RNIC sends the completion information in RDMA VP to the remote host Host using RDMA Send, indicating that the data reading process is completed.

[0041] In the embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0042] In addition, all functional modules in the embodiments of the present invention may be integrated into one processor, or each module may be a separate device, or two or more modules may be integrated into one device; each functional module in the embodiments of the present invention may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0043] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by program instructions and related hardware. The aforementioned program instructions can be stored in a computer-readable storage medium. When the program instructions are executed, the steps of the above method embodiment are executed; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM), a magnetic disk or an optical disk, and other media that can store program codes.

[0044] It should be understood that the use of "system", "device", "unit" and / or "module" in this application is only a method for distinguishing different components, elements, parts, parts or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0045] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0046] If a flow chart is used in the present application, the flow chart is used to illustrate the operations performed by the system according to the embodiment of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, each step can be processed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or a certain step or several steps of operations can be removed from these processes.

[0047] The above is a detailed introduction to a NVMeoF hardware unloading device and method provided by the present invention. The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in the field, and the general principles defined in this article can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in this article, but will conform to the widest scope consistent with the principles and novel features disclosed in this article.

Claims

1. A hardware unloading device for NVMeoF, characterized in that: include: Hardware offload engine, cache module, network interface control module and NVMe storage module; The cache module is connected to the hardware offload engine and the network interface control module respectively; The network interface control module is connected to the remote host end; The hardware offload engine is connected to the NVMe storage module via a PCIe interface; The hardware offload engine is used to sequentially perform encapsulation, format checking, address translation, reverse address mapping, direct access and management data transmission operations on remote host-side NVMeoF commands; The cache module is used to cache the RDMA read message and RDMA write message of the network interface control module, the NVMe command after address conversion and reverse address mapping, the read operation data and the write operation data; The network interface control module is used to extract and process virtual descriptors, establish descriptor extraction tasks, cache root endpoint mode requests, construct request messages and response messages, verify request messages and convert formats, and verify and cache response messages; The network interface control module is also used to encapsulate and construct ROCE messages, submit completion messages, control transmission rates, configure transmission parameters, and perform network communications with remote host terminals.

2. The NVMeoF hardware offloading device according to claim 1, wherein: The hardware offload engine includes a VP queue processing unit, an NVMeoF command analysis mapping unit, an arbitration processing unit, an NVMeoF command processing unit, a data transmission unit and a PCIe EP unit; The VP queue processing unit is used to interact with the network interface control module, receive the NVMeoF command package of the remote host, send the NVMeoF command to the NVMeoF submission queue, and send the corresponding response package to the remote host after the access is completed; The NVMeoF command analysis and mapping unit is used to extract the NVMeoF command from the NVMeoF submission queue, check whether the NVMeoF command conforms to a preset format, and if so, perform address conversion processing to form the NVMe command, and store it in the submission queue of the NVMe storage module; The NVMeoF command analysis and mapping unit is further used to perform reverse address mapping on the submission queue of the NVMe storage module to obtain the NVMe command and store it in the submission queue of the NVMeoF; The arbitration processing unit is used to process the NVMe command and then send it to the NVMeoF queue; The NVMeoF command processing unit is used to process the corresponding NVMeoF queue and interact with the NVMe storage module; The data transmission processing unit is used to manage the data transmission between the hardware offload engine and the cache module.

3. The NVMeoF hardware offloading device according to claim 2, wherein: The cache module includes: a VP cache unit, an NVMe storage module queue cache unit, an NVMeoF queue cache unit and a read-write data cache unit; The VP cache unit is used to cache the RDMA read message and the RDMA write message in the network interface control module; The NVMe storage module queue cache unit is used to cache the NVMe commands after address conversion by the NVMeoF command analysis mapping unit; The NVMeoF queue cache unit is used to cache the NVMe commands after the NVMeoF command analysis mapping unit undergoes reverse address mapping; The read-write data cache unit is used to cache the read operation data and the write operation data generated when the RDMA in the network interface control module performs the read operation and the write operation.

4. The NVMeoF hardware offloading device according to claim 3, characterized in that: The network interface control module includes: a descriptor communication processing unit, a reliable transmission control unit, a message construction unit, a message processing unit and an address conversion unit; The descriptor communication processing module is used to extract the request descriptor and the response descriptor from the VP queue processing unit, and to establish a descriptor extraction task based on the retransmission descriptor trigger after scheduling, caching, parsing and checking in sequence; The reliable transmission control unit is used to cache the root endpoint mode request before the root endpoint mode confirmation is completed; The message construction unit is used to construct a request message and / or a response message according to the request descriptor and / or the response descriptor; The message processing unit is used to verify the request message and filter the request message; The message processing unit is further used to check the response message, determine whether it needs to be retransmitted, and transmit the response message that needs to be retransmitted to the read-write data cache unit; The address conversion unit is used to convert the NVMeoF reaction cabin component into the RDMA VP format.

5. The NVMeoF hardware offloading device according to claim 4, characterized in that: The network interface control module further includes: a message construction and parsing unit, a CQE processing unit, a congestion control unit, a configuration processing unit and a network interface; The message construction and parsing unit is used to perform a legality check on the RDMA-based Ethernet input message and to encapsulate and construct the RDMA-based Ethernet output message; The CQE processing unit is used to submit the completion message of the inspection or packaging and construction; The congestion control unit is used to control the transmission rate of the network communication of the remote host end; The configuration processing unit is used to set and store the transmission parameter configuration between the CPU of the remote host and the network interface control module; The network interface is used for network communication with the remote host.

6. A method for hardware unloading of NVMeoF, implemented based on the hardware unloading device of NVMeoF according to any one of claims 1 to 5, characterized in that: The NVMeoF hardware offloading method includes a write operation of NVMeoF based on RDMA, which is specifically the following steps: The remote host encapsulates the NVMeoF command into an RDMA message and sends it to the network interface control module; The network interface control module completes the unloading of the RDMA protocol and strips the NVMeoF command encapsulation from the RDMA message; Writing the NVMeoF command encapsulation into a submission queue of a cache module; The hardware offload engine takes out the NVMeoF command encapsulation in the submission queue of the cache module, and completes the protocol offload work of command parsing and address conversion.

7. The NVMeoF hardware uninstallation method according to claim 6, characterized in that: After completing the protocol unloading work of command parsing and address translation, the following steps are also included: The hardware offload engine allocates data space of the read and write data cache unit in the cache module, and writes the RDMA read message into the VP queue processing unit; The remote host reads the data from the memory according to the RDMA read request initiated by the network interface control module, writes the read data into the data space of the read-write data cache unit allocated in the cache module, and notifies the hardware offload engine; The NVMe storage module retrieves the data to be written from the data space of the read-write data cache unit allocated in the cache module according to the doorbell signal sent by the hardware offload engine; After the data transmission is completed, the NVMe storage module writes the completion command into the NVMeoF queue cache unit in the cache module, which is mapped and converted by the hardware offload engine, encapsulated as an NVMeoF reaction cabin component, and written into the NVMeoF completion queue in the cache module; The network interface control module converts the NVMeoF reaction cabin component in the NVMeoF completion queue into the RDMA VP format; The network interface control module sends the completion information in the VP queue processing unit to the remote host end.

8. The NVMeoF hardware uninstallation method according to claim 6, wherein: The hardware offloading method of NVMeoF includes a read operation of NVMeoF based on RDMA, which is specifically the following steps: The remote host encapsulates the NVMeoF command into an RDMA message and sends it to the network interface control module; The network interface control module completes the unloading of the RDMA protocol and strips the NVMeoF command encapsulation from the RDMA message; Writing the NVMeoF command encapsulation into a submission queue of a cache module; The hardware offload engine takes out the NVMeoF command encapsulation in the submission queue of the cache module and completes the protocol offload work of command parsing and address conversion; The NVMe storage module takes out the NVMe command from the submission queue of the NVMe storage module of the cache module according to the doorbell signal sent by the hardware offload engine, and writes the data into the read-write data cache unit in the cache module after execution; After the data transmission is completed, the NVMe storage module writes the completion command into the NVMeoF queue cache unit in the cache module, which is mapped and converted by the hardware offload engine, encapsulated as an NVMeoF reaction cabin component, and written into the NVMeoF completion queue in the cache module; The network interface control module initiates an RDMA write request to write the data in the read-write data cache unit in the cache module into the remote host memory; The network interface control module converts the NVMeoF reaction cabin component into an RDMA VP format; The network interface control module sends the completion information in the VP queue processing unit to the remote host end.

Images