Nvmeof remote storage system and method based on gpu direct command control

Abstract

This invention discloses an NVMeoF remote storage system and method based on direct GPU command control. It clearly divides the I / O processing tasks in the NVMeoF environment into two main modules: "GPU processing NVMe commands" and "CPU processing network protocols." A doorbell notification mechanism enables collaboration between the two modules, allowing the CPU to focus on its strength in network protocol processing and the GPU on its strengths in NVMe command generation and data processing. This significantly reduces CPU load and establishes an end-to-end "zero-copy" data path from remote NVMe storage to GPU memory, eliminating unnecessary intermediate data copies and maximizing data throughput. This invention enables near-local, low-latency, high-bandwidth access to remote NVMe storage devices by the GPU within the NVMeoF environment.

Description

Technical Field

[0001] This invention relates to distributed computing technology, specifically to an NVMeoF remote storage system and method based on GPU direct command control. Background Technology

[0002] In distributed computing scenarios based on NVMe over Fabrics (NVMeoF) (such as AI training and high-performance computing), existing technologies cannot simultaneously achieve low-latency storage access and low CPU overhead. Specifically, when the GPU needs to frequently access remote storage devices, the traditional CPU-centric I / O path becomes a system performance bottleneck, with the following drawbacks:

[0003] Excessive data copying: Unnecessary memory copies occur in the I / O path, consuming valuable memory bandwidth and PCIe bandwidth.

[0004] Frequent CPU intervention: Each I / O request requires multiple CPU interventions (command encapsulation, data transfer, and processing), resulting in high CPU utilization and difficulty in supporting high-concurrency I / O.

[0005] GPU computing resources waiting for I / O: Due to high I / O latency, the GPU often needs to wait for data, and its powerful computing capabilities are idle, reducing the computing efficiency of the entire system.

[0006] Existing technologies such as GPU pass-through storage cannot be directly applied to network storage scenarios, limiting the scalability and flexibility of high-performance computing clusters.

[0007] To address the aforementioned issues, existing research has proposed the BaM (Big Accelerator Memory) architecture.

[0008] BaM's structure consists primarily of a GPU, a local NVMe SSD, a host CPU, and memory. Its core function is to allocate and manage NVMe commit queues (SQ) and completion queues (CQ) within the GPU's video memory.

[0009] BaM workflow (methods and steps):

[0010] Initialization: The CPU assists in creating NVMe queues in the GPU memory and configures the NVMe SSD controller to recognize these queues.

[0011] Command submission: The GPU computing core directly constructs the NVMe command in the SQ located in the video memory, and then writes it to the SSD's doorbell register via PCIe to notify the SSD of the new command.

[0012] Data transfer: NVMe SSDs directly read commands in SQ via PCIe and then write the data directly to a specified location in the GPU memory via DMA.

[0013] Completion notification: The SSD writes the completion status to the CQ located in the GPU's video memory, and the GPU learns that the I / O is complete by polling the CQ.

[0014] The core of BaM is that it enables the GPU to directly control the local NVMe device, with data flowing directly from the SSD to the GPU memory, completely bypassing the CPU and system memory.

[0015] Because the BaM architecture is designed for native PCIe environments, its mechanism of directly controlling NVMe queues via the GPU cannot directly handle NVMeoF commands that require network protocol encapsulation / decapsulation. Therefore, in NVMeoF scenarios, if the BaM concept is forcibly applied, the GPU needs to have the ability to process the network protocol stack, which is currently extremely costly and impractical, preventing the advantages of BaM from being extended to the network storage field. Summary of the Invention

[0016] The technical problem to be solved by the present invention is to provide an NVMeoF remote storage system and method based on GPU direct command control, which can realize near-local low-latency, high-bandwidth access of the GPU to remote NVMe storage devices in the NVMeoF environment.

[0017] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0018] A method for NVMeoF remote storage based on GPU direct command control is disclosed. The method is applied to an NVMeoF remote storage system comprising an initiating host system and a remote target system. The initiating host system includes a GPU domain, a CPU domain, and a first RDMA network interface card (NIC). The remote target system includes a local NVMe SSD, NVMe-oF target software, and a second RDMA NIC. The method includes the following steps:

[0019] The application in the GPU domain initiates a read request, generates the corresponding NVMe command in the GPU memory through the GPU computing core, adds it to the commit queue SQ, and then notifies the CPU domain through a doorbell.

[0020] The CPU domain encapsulates the NVMe commands in the submission queue SQ into NVMe over RDMA command frames and submits an RDMA Send request to the first RDMA network card.

[0021] The first RDMA network card packages the corresponding NVMe over RDMA command frame into a network data packet according to the RDMA Send request and sends it to the second RDMA network card;

[0022] The second RDMA network card receives network data packets. The NVMe-oF target software parses the NVMeover RDMA command frame from the network data packets to obtain the corresponding NVMe command, and submits it to the local NVMe SSD to perform a read operation to obtain the data in the local NVMe SSD.

[0023] The NVMe-oF target software uses RDMA Write operation on the second RDMA network card and the first RDMA network card according to the information of the NVMe over RDMA command frame to send the data in the local NVMe SSD to the GPU memory. After the data is sent, a completion message is sent to the CPU domain.

[0024] The CPU domain generates a completion event and adds it to the completion queue (CQ) in the GPU memory. The GPU domain polls the completion queue (CQ) through the GPU computing core. When the completion event is detected, the read operation is confirmed as successful, and the application is notified.

[0025] Furthermore, when the CPU domain is notified via the doorbell, the GPU computing core performs a write operation on the doorbell register to trigger a CPU interrupt, causing the CPU domain to read NVMe commands from the GPU memory's commit queue (SQ) after the interrupt.

[0026] Furthermore, the doorbell register is an MMIO register mapped to the CPU address space.

[0027] Furthermore, when the GPU computing core generates the corresponding NVMe command in the GPU memory and adds it to the commit queue SQ, it includes:

[0028] A corresponding NVMe command is constructed at the tail of the commit queue SQ in the GPU memory. The opcode field OPC of the NVMe command is set to read operation, the start logical block address field SLBA is set to the start logical block address corresponding to the read request, the block number field NLB is set to the length of the data requested to be transferred in the read request, and the data pointer field DPTR is set to the address of the data buffer in the GPU memory and the remote access key RKey corresponding to the data buffer. After the fields of the NVMe command are filled, the tail pointer of the commit queue SQ is updated.

[0029] Furthermore, when the NVMe-oF target software performs RDMA Write operations on the second RDMA network card and the first RDMA network card based on the information in the NVMe over RDMA command frame, it specifically includes:

[0030] The NVMe-oF target software sends the address of the data buffer in the data pointer field DPTR and the remote access key RKey to the second RDMA network card;

[0031] The second RDMA network card directly writes the data in the local NVMe SSD to the first RDMA network card through RDMA Write, based on the address of the data buffer and the remote access key RKey.

[0032] The first RDMA network card directly DMA-transfers data from the local NVMe SSD to the data buffer of the GPU's video memory, allowing data transfer in the local NVMe SSD to completely bypass the CPU and system memory at the initiating end.

[0033] Furthermore, before the application initiates a read request, the process includes: the CPU domain registering a data buffer in the GPU memory via an RDMA driver, obtaining a remote access key RKey corresponding to the data buffer, and providing the remote access key RKey and the data buffer in the GPU memory to the GPU domain for constructing the NVMe command.

[0034] Furthermore, the GPU domain and the CPU domain exchange information through shared command descriptors located in the GPU memory. These shared command descriptors include, but are not limited to, the addresses of NVMe commands in the GPU memory, the remote access key RKey for the GPU data buffer, and the addresses of data buffers in the GPU memory.

[0035] Furthermore, when the GPU domain polls the completion queue CQ through the GPU computing core, the process includes: the GPU computing core periodically checking whether the head pointer and tail pointer of the completion queue CQ are equal; when it detects that the head pointer and tail pointer are not equal, it reads the completion event from the completion queue CQ, updates the head pointer of the completion queue CQ, and notifies the application that the read operation is complete.

[0036] Furthermore, the method also includes a step of performing a write operation on the remote target, specifically including:

[0037] The application in the GPU domain initiates a write request, generates a corresponding NVMe write command in the GPU memory through the GPU computing core, and adds it to the commit queue SQ. The data pointer field of the NVMe write command includes the GPU memory address of the data to be written and the remote access key RKey. Then, the CPU domain is notified through a doorbell.

[0038] The CPU domain encapsulates the NVMe write command in the submission queue SQ into an NVMe over RDMA command frame and submits an RDMA Send request to the first RDMA network card.

[0039] The first RDMA network card packages the corresponding NVMe over RDMA command frame into a network data packet according to the RDMA Send request and sends it to the second RDMA network card;

[0040] The second RDMA network card receives network data packets. The NVMe-oF target software parses the NVMeover RDMA command frame from the network data packets to obtain the corresponding NVMe write command and submits it to the local NVMe SSD.

[0041] The NVMe-oF target software uses RDMA Read operation to read the data to be written from the GPU memory of the initiating end according to the GPU memory address and remote access key RKey, and writes the data to be written to the local NVMe SSD. After the writing is completed, the completion message is sent to the CPU domain through the second RDMA network card and the first RDMA network card in sequence.

[0042] The CPU domain generates a completion event and adds it to the completion queue (CQ) in the GPU memory. The GPU domain polls the completion queue (CQ) through the GPU computing core. When the completion event is detected, the write operation is confirmed as successful, and the application is notified.

[0043] This invention also proposes an NVMeoF remote storage system based on GPU direct command control, comprising an initiating host system and a remote target system. The initiating host system includes a GPU domain, a CPU domain, and a first RDMA network card. The remote target system includes a local NVMe SSD, NVMe-oF target software, and a second RDMA network card. The NVMeoF remote storage system is programmed or configured to execute the steps of the NVMeoF remote storage method based on GPU direct command control.

[0044] Compared with the prior art, the advantages of the present invention are as follows:

[0045] Because this invention offloads NVMe command processing tasks from the CPU to the GPU and enables direct data transfer (zero copy) from remote storage to GPU memory, it significantly reduces CPU intervention and the number of data copies. This inevitably leads to a significant reduction in CPU load and a shortening of I / O path latency.

[0046] Because GPUs can directly and quickly generate and process NVMe commands without waiting for CPU scheduling and processing, they acquire data much faster. This inevitably reduces the amount of GPU computing resources idle due to waiting for I / O, thereby improving the overall system's computing efficiency and throughput.

[0047] Because this invention retains the mature advantage of CPUs in handling complex network protocols while introducing direct GPU control over NVMe queues, it successfully extends the high-efficiency concept of the BaM architecture to the NVMeoF network environment. This inevitably means that this invention can effectively solve the technical challenges of existing BaM technology in handling network storage scenarios, providing a better solution for distributed high-performance computing. Attached Figure Description

[0048] Figure 1 This is the overall system architecture of an embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of the complete NVMeoF process of CPU-GPU collaboration in the method of this embodiment of the invention.

[0050] Figure 3 This is a schematic diagram of the GPU manipulating the NVMe queue process in the method of this embodiment of the invention.

[0051] Figure 4 This is a code example of a shared command descriptor in an embodiment of the present invention. Detailed Implementation

[0052] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0053] Before introducing specific embodiments of the present invention, relevant concepts or terms will be explained.

[0054] NVMe (Non-Volatile Memory Express): A high-performance, low-latency solid-state drive interface protocol. Its core mechanism lies in supporting multiple queues (up to 64K queues), deep parallelism (each queue is 64K deep), and a reduced instruction set. It connects directly to the CPU via the PCIe bus to fully utilize the performance of the SSD.

[0055] NVMe over Fabrics (NVMeoF): A network protocol used to extend NVMe functionality over a network. It encapsulates local NVMe commands and responses into network frames (such as RDMA, TCP) for transmission, allowing hosts to access remote storage devices as if they were local NVMe devices.

[0056] RDMA (Remote Direct Memory Access): A network technology that allows one computer to directly access the memory of another computer without the intervention of the other computer's operating system. Its core mechanisms include kernel bypass (applications communicate directly with the network card), zero-copy (the network card directly reads and writes application memory), and memory registration (pre-locking memory for direct access by the network card).

[0057] GPU Direct Storage (GDS): A technology designed to establish a direct data path from NVMe storage to GPU memory, aiming to reduce data copying in system memory, thereby reducing latency and freeing up the CPU.

[0058] SQ (Submission Queue) / CQ (Completion Queue): In the NVMe protocol, these are queues used by the host to submit commands to the controller and by the controller to report command completion status to the host, respectively.

[0059] Doorbell Register: In the NVMe protocol, the host notifies the controller that there are new commands pending in the SQ by writing to this register.

[0060] Example 1

[0061] In an NVMeoF environment, enabling GPUs to access remote NVMe storage devices requires addressing the following three specific technical challenges:

[0062] Issue 1: The problem of long I / O path and high latency when GPU accesses remote NVMe storage: Data needs to go through multiple transfers and copies of "remote storage → target memory → initiating CPU memory → GPU memory", resulting in high transmission latency.

[0063] Question 2: The problem of excessive CPU load when processing network protocol stack and NVMe commands: The CPU needs to be responsible for complex network protocol (such as RDMA) encapsulation and decapsulation and NVMe command processing at the same time. In high-throughput scenarios, CPU resources are quickly exhausted, becoming a system bottleneck.

[0064] Question 3: Existing GPU direct storage technology cannot be effectively applied to network storage environments: Optimization solutions such as BaM are designed for local NVMe devices, and their mechanism of allowing the GPU to directly control the NVMe queue is difficult to apply directly to NVMeoF environments that require network protocol conversion.

[0065] The root cause of problems 1 and 2 lies in the unclear division of responsibilities within the architecture. Current technology imposes both network protocol processing and NVMe command processing—two computationally intensive tasks—on the CPU, while the GPU, as the final data consumer, remains passive in the I / O control path. This architecture leads to coupling between the control and data paths, making end-to-end optimization impossible.

[0066] The technical reason for question 3 lies in the gap between the local PCIe architecture and the network protocol stack. GPUs excel at parallel computing and direct memory access, but lack the ability to handle complex network protocols (such as TCP / IP or RDMA Verbs). Directly extending the GPU pass-through memory architecture to the network environment requires the GPU to understand and generate network packets, which is impractical under the current hardware and software ecosystem.

[0067] To address the aforementioned issues, this embodiment proposes an NVMeoF remote storage method based on direct GPU command control. This method enables near-local, low-latency, high-bandwidth access to remote NVMe storage devices by the GPU within the NVMeoF environment. It also achieves efficient division of labor between the CPU and GPU on the I / O path, allowing the CPU to focus on its strengths in network protocol processing and the GPU on its strengths in NVMe command generation and data processing, thereby significantly reducing CPU load. Furthermore, it constructs an end-to-end "zero-copy" data path from remote NVMe storage to GPU memory, eliminating unnecessary intermediate data copies and maximizing data throughput.

[0068] The method in this embodiment is applied to Figure 1 The diagram shows the NVMeoF remote storage system, including the initiating host system and the remote target system. The components of the initiating (compute node) and the target (storage node) systems and their collaborative relationships are as follows:

[0069] The initiating host system adopts a heterogeneous division of labor architecture, and its detailed components include:

[0070] GPU Domain: Located within the initiator, it is the core control domain. This domain contains:

[0071] CUDA applications: the starting point for initiating storage requests, passing I / O requests down to the next layer via the cuFile API;

[0072] GDS driver layer: responsible for receiving file-level requests from the application layer and converting them into block-level I / O operations;

[0073] NVMe command generation and parsing module: Firmware / driver logic deployed inside the GPU directly constructs command entries conforming to the NVMe specification in the GPU memory. Crucially, in the NVMe commands generated by this module, the data pointer DPTR field is directly filled with the GPU memory's device address and corresponding RKey, rather than the system memory address used in traditional solutions.

[0074] NVMe SQ & CQ in GPU memory: NVMe commit queues (SQ) and completion queues (CQ) are directly allocated in the reserved area of ​​GPU memory. SQ stores pending NVMe commands generated by the GPU, while CQ receives the completion status of remote SSD writes.

[0075] CPU domain: Responsible for all network protocol processing, including:

[0076] CPU network protocol stack: Implements the NVMe over Fabrics protocol layer, responsible for encapsulating NVMe commands into the network transport layer or decapsulating received network frames;

[0077] RDMA driver: Manages RDMA network cards.

[0078] The first RDMA network card (HCA) acts as a hub connecting the GPU, CPU, and physical network, and is responsible for performing RDMA operations.

[0079] The remote target receives an RDMA data packet from the network Fabric, which includes:

[0080] Second RDMA network card: Receives RDMA operations from the initiating end;

[0081] NVMe-oF target software: parses the NVMe over RDMA protocol and restores network frames to local NVMe commands;

[0082] Local NVMe SSD: Performs the actual storage I / O operations and, based on the DPTR information in the command, writes the data directly back to the initiating GPU memory via RDMAWrite.

[0083] also, Figure 1 The "doorbell notification" highlighted in the code is a key interface for implementing CPU-GPU collaboration. After the GPU constructs the NVMe command in its video memory SQ (Segment Query), it triggers a CPU interrupt by writing to the doorbell register. This doorbell register is mapped into the CPU's address space; a write operation wakes up the CPU to fetch the command from the GPU's video memory and perform network encapsulation. In this embodiment, the doorbell register is an MMIO register mapped to the CPU's address space, and a write operation triggers a CPU interrupt. The GPU notifies the CPU through this interface. A code example is shown below:

[0084] gpu_doorbell_write(struct io_descriptor *desc);

[0085] The core of this architecture lies in separating responsibilities, allowing the GPU and CPU to handle the tasks they are best at, and achieving efficient collaboration through RDMA network cards and doorbell notification mechanisms.

[0086] For the system architecture described above, the read operation flow of the method in this embodiment is as follows: Figure 2 As shown, it includes the following steps:

[0087] Phase 1 (Control Flow Initiation):

[0088] The application in the GPU domain initiates a read request, generates the corresponding NVMe command in the GPU memory through the GPU computing core, adds it to the commit queue SQ, and then notifies the CPU domain through a doorbell.

[0089] The CPU domain encapsulates the NVMe commands in the submission queue SQ into NVMe over RDMA command frames and submits an RDMA Send request to the first RDMA network card.

[0090] The first RDMA network card packages the corresponding NVMe over RDMA command frame into a network data packet according to the RDMA Send request and sends it to the second RDMA network card;

[0091] Phase Two (Execution on Storage Side):

[0092] The second RDMA network card receives network data packets. The NVMe-oF target software parses the NVMeover RDMA command frame from the network data packets to obtain the corresponding NVMe command, and submits it to the local NVMe SSD to perform a read operation to obtain the data in the local NVMe SSD.

[0093] Phase 3 (Direct Data Stream Transmission):

[0094] The NVMe-oF target software uses RDMA Write operation on the second RDMA network card and the first RDMA network card according to the information of the NVMe over RDMA command frame to send the data in the local NVMe SSD to the GPU memory. After the data is sent, a completion message is sent to the CPU domain.

[0095] Phase Four (Completion Notification):

[0096] The CPU generates a completion event and adds it to the completion queue (CQ) in the GPU's video memory. After writing the completion status to the GPU's video memory CQ, the CPU notifies the GPU through the completion event callback interface. A code example is shown below:

[0097] gpu_cq_completion_notify(uint16_t command_id);

[0098] The GPU domain polls the completion queue (CQ) through the GPU computing core. When the completion event is detected, the read operation is confirmed as successful, and the application is notified.

[0099] In each of the above stages, the GPU replaces the CPU as the generator and manager of NVMe commands, and directly manipulates the NVMe queue. Figure 3 This demonstrates the complete lifecycle of GPU direct control over NVMe queues. The entire command construction process is completed within the GPU domain, with the CPU only notified via a doorbell mechanism when network transmission is required. This decouples NVMe command processing from network protocol processing. The specific process is as follows:

[0100] S101 (Command Construction and Submission): The GPU compute core directly constructs the NVMe command in its own memory's submission queue. Crucially, the data pointer uses the GPU memory address and RKey. Subsequently, it notifies the CPU to send an RDMA network message by writing to the doorbell register. In this step, the GPU directly operates on the NVMe queue in its own memory, without CPU intervention in command construction. The DPTR field in the command directly points to the GPU memory and carries the RKey, providing routing information for subsequent remote SSD direct writing to the GPU memory. Specifically, as shown... Figure 3 As shown, perform the following operations:

[0101] 1. Construct an NVMe command in the SQ empty space of the GPU memory. Key fields include:

[0102] OPC (Opcode): Set to read opcode;

[0103] SLBA (Start Logical Block Address): The LBA address obtained by converting the file offset;

[0104] NLB (Number of Logical Blocks): The length of the data requested for transmission;

[0105] DPTR (Data Pointer): Set to the physical address of the GPU data buffer in video memory and the RKey (Remote Access Key) of the buffer;

[0106] 2. Update the local SQ tail pointer to mark the new command as ready;

[0107] 3. The GPU updates the doorbell register via PCIe MMIO write operation, triggering a CPU interrupt and notifying the CPU that there are new NVMe commands to be sent.

[0108] S102 (SSD Fetch and Execute Command): The SSD actively fetches the command and, according to the instructions in the command, writes the data directly to the GPU's video memory data buffer via RDMA. Upon completion, it enters the write status in the completion queue. In this step, the data path completely bypasses the initiating CPU and system memory, achieving true zero-copy processing from "storage → network → GPU memory". Specifically, as shown... Figure 3 As shown, perform the following operations:

[0109] 1. The SSD periodically retrieves NVMe commands from the SQ at the initiating end. In network scenarios, the SQ content has been encapsulated in the NVMe-oF command and transmitted to the target end.

[0110] 2. The DPTR field in the SSD parsing command is used to identify the target address as the GPU memory address of the initiating end, and the RKey is extracted;

[0111] 3. The SSD reads data from the local machine. The target end initiates an RDMA Write operation through the RDMA network card, using the GPU memory address and RKey in the DPTR to directly transfer the data to the GPU memory data buffer through the RDMA network card of the initiating end.

[0112] 4. After the data transmission is complete, write the completion queue entry to the CQ in the GPU memory of the initiating end.

[0113] S103 (GPU Processing Completion Status): The GPU detects I / O completion through polling, reads the completion status, and confirms it. This entire process forms a highly efficient local I / O closed loop led by the GPU. In this step, the entire I / O closed loop (command submission → data transfer → completion confirmation) is completed by the GPU; the CPU is only briefly awakened in step S101 via a doorbell mechanism to handle network encapsulation, greatly reducing CPU load. Specifically, as shown... Figure 3 As shown, perform the following operations:

[0114] 1. The GPU computing core periodically checks whether the CQ head pointer and CQ tail pointer are equal. If they are not equal, it indicates that a new completion entry has been created;

[0115] 2. The GPU reads the completed entry from the CQ and parses the command completion status. If the status code is success, it confirms that the data is ready. At this time, the GPU updates the CQ header pointer, confirms to the SSD that the processing of the entry has been completed, and releases the CQ slot;

[0116] 3. The GPU notifies the CUDA application that I / O is complete, and the application can directly access the data in the video memory.

[0117] The following section provides a detailed description of each stage of the method in this embodiment, based on the above content.

[0118] Before Phase 1, an initialization process is performed, including: the CPU domain registers the data buffer in the GPU memory through the RDMA driver, obtains the remote access key RKey corresponding to the data buffer, and provides the remote access key RKey and the data buffer in the GPU memory to the GPU domain through a shared command descriptor to construct the NVMe command.

[0119] In this embodiment, the shared command descriptor is located in the GPU memory and is accessed by both the CPU and GPU. It is used to pass the network encapsulation context for a single I / O request, such as... Figure 4 As shown, the structure of a shared command descriptor includes:

[0120] gpu_sq_entry_addr: Points to the storage location of NVMe commands in the GPU memory;

[0121] rkey: is the remote access key RKey for the data buffer;

[0122] gpu_data_addr: The physical address of the GPU memory in the data buffer.

[0123] Through the above initialization, the GPU domain obtains the RKey and address information required to construct the NVMe command.

[0124] like Figure 2 As shown, Phase 1 (Control Flow Initiation) demonstrates how, after the GPU generates NVMe commands, it "wakes up" the CPU via a doorbell mechanism, with the CPU then responsible for subsequent network protocol encapsulation and transmission. This embodies the core principle of "GPU handling NVMe, CPU handling network." The read operation flow includes the following steps:

[0125] Step S201: The CUDA application in the GPU domain initiates a read request (such as calling the cuFileRead API) to request to read data from the remote NVMe SSD.

[0126] Step S202: The GPU computing core responds to the read request and constructs the corresponding NVMe read command in the GPU memory commit queue SQ. The specific construction process includes:

[0127] Build the corresponding NVMe command at the tail of the GPU memory commit queue SQ;

[0128] Set the opcode field OPC of the NVMe command to read operation;

[0129] Set the Start Logical Block Address (SLBA) field to the start logical block address corresponding to the read request. The start logical block address is calculated based on the file offset of the read request.

[0130] Set the block number field NLB to the number of logical blocks corresponding to the length of data requested to be transmitted in the read request;

[0131] Set the data pointer field DPTR to the address of the data buffer in the GPU memory and the remote access key RKey corresponding to the data buffer.

[0132] Step S203: After the fields of the NVMe command are filled in, update the tail pointer of the commit queue SQ to mark the new command as ready.

[0133] Step S204: The GPU computing core executes a doorbell notification operation, triggering a CPU domain interrupt. Specifically, the GPU performs an MMIO write operation on the doorbell register. The write operation triggers an interrupt, waking up the CPU interrupt handler.

[0134] Step S205: The CPU domain responds to the interrupt, directly accesses the GPU memory through the information in the shared command descriptor, and reads the NVMe command from SQ.

[0135] Step S206: The CPU domain encapsulates the NVMe command into an NVMe over RDMA command frame. During the encapsulation process, the GPU memory address and RKey information in the DPTR field are retained for subsequent use by the remote target.

[0136] Step S207: The CPU domain submits an RDMA Send request to the first RDMA network card. The first RDMA network card packages the NVMe overRDMA command frame into a network data packet and sends it to the second RDMA network card through the Fabric network.

[0137] like Figure 2 As shown, Phase Two (Storage-side Execution): The remote target processes commands according to standard procedures. The read operation process includes the following steps:

[0138] Step S208: The second RDMA network card receives network data packets, and the NVMe-oF target software parses the NVMe over RDMA command frame from the network data packets to restore the original NVMe read command.

[0139] Step S209: The NVMe-oF target software submits the parsed NVMe command to the local NVMe SSD. The SSD performs a read operation, reading the data from the flash memory to the target memory buffer.

[0140] like Figure 2As shown, Phase 3 (direct data transfer): This is the performance-critical path. The target end uses the GPU memory address and RKey obtained from the command to directly send data to the initiating end's GPU memory via RDMA Write. This path completely bypasses the initiating end's CPU and system memory, achieving true "zero copy". The read operation process includes the following steps:

[0141] Step S210: The NVMe-oF target software parses the DPTR field of the NVMe command and extracts the address of the data buffer in the GPU memory and the remote access key RKey contained therein.

[0142] Step S211: The NVMe-oF target software instructs the second RDMA network card to perform an RDMA Write operation. The second RDMA network card initiates an RDMA Write request through the Fabric network based on the address of the data buffer and the RKey.

[0143] Step S212: The first RDMA network card receives the RDMA Write request and uses its DMA engine to directly DMA transfer the SSD data to the GPU's video memory data buffer. During this process, the data transfer completely bypasses the initiating CPU and system memory, achieving true zero-copy.

[0144] Step S213: After the data transmission is completed, the target end sends a completion message (RDMA Send) to the initiating end through the second RDMA network card.

[0145] like Figure 2 As shown, in Phase Four (Completion Notification): Control flow returns to the CPU and GPU. The CPU processes the network completion message, and the GPU finally confirms I / O completion. The read operation flow includes the following steps:

[0146] Step S214: The first RDMA network card receives the completion message, and the CPU domain generates a completion event.

[0147] Step S215: The CPU domain writes the completion event to the completion queue (CQ) in the GPU memory. Specifically, the CPU writes the completion status (success / failure) and related metadata to the CQ in the GPU memory.

[0148] Step S216: The GPU domain polls the completed queue CQ through the GPU computing core. The polling process includes:

[0149] The GPU computing core periodically checks the head pointer and tail pointer of CQ. When it detects that the head pointer and tail pointer are not equal, it indicates that there is a new completed entry.

[0150] The GPU reads the completion event from the CQ, parses the completion status, and updates the CQ head pointer to confirm that the entry has been processed.

[0151] When the completion status is successful, the CUDA application is notified that the read operation was successful and the application can directly access the data in the GPU memory.

[0152] The write operation process and operation process of the method in this embodiment are basically the same, including the following steps:

[0153] Phase 1 (Control Flow Initiation):

[0154] The application in the GPU domain initiates a write request, generates a corresponding NVMe write command in the GPU memory through the GPU computing core, and adds it to the commit queue SQ. The data pointer field of the NVMe write command includes the GPU memory address of the data to be written and the remote access key RKey. Then, the CPU domain is notified through a doorbell.

[0155] The CPU domain encapsulates the NVMe write command in the submission queue SQ into an NVMe over RDMA command frame and submits an RDMA Send request to the first RDMA network card.

[0156] The first RDMA network card packages the corresponding NVMe over RDMA command frame into a network data packet according to the RDMA Send request and sends it to the second RDMA network card;

[0157] Phase Two (Execution on Storage Side):

[0158] The second RDMA network card receives network data packets. The NVMe-oF target software parses the NVMeover RDMA command frame from the network data packets to obtain the corresponding NVMe write command and submits it to the local NVMe SSD.

[0159] Phase 3 (Direct Data Stream Transmission):

[0160] The NVMe-oF target software uses RDMA Read operation to read the data to be written from the GPU memory of the initiating end according to the GPU memory address and remote access key RKey, and writes the data to be written to the local NVMe SSD. After the writing is completed, the completion message is sent to the CPU domain through the second RDMA network card and the first RDMA network card in sequence.

[0161] Phase Four (Completion Notification):

[0162] The CPU domain generates a completion event and adds it to the completion queue (CQ) in the GPU memory. The GPU domain polls the completion queue (CQ) through the GPU computing core. When the completion event is detected, the write operation is confirmed as successful, and the application is notified.

[0163] Since the write and read operation processes are essentially the same in each stage, the only difference is that in a write operation, the data flow is from the GPU memory at the initiating end to the remote SSD, which is the opposite of the read operation; and according to the NVMe over Fabrics protocol principle and RDMA operation characteristics, the target end should use RDMA Read operation to pull data from the initiating end in a write operation. The execution steps of each stage of the read operation process have been described in detail above, so the execution steps of each stage of the write operation process will not be repeated here.

[0164] In summary, this invention proposes a high-efficiency, low-latency NVMeoF remote storage system based on direct GPU command control, and a method for NVMeoF remote storage based on direct GPU command control within this system architecture. The solution utilizes a CPU-GPU hybrid computing system in an NVMe over Fabrics environment. Key points are as follows:

[0165] Separation of Duties Architecture: The I / O processing tasks in the NVMeoF environment are clearly divided into two major modules: "GPU processing NVMe commands" and "CPU processing network protocols," and the two are coordinated through a doorbell notification mechanism. This is the core invention concept.

[0166] NVMe queues in GPU memory: This section describes the specific implementation methods for allocating and managing NVMe commit queues (SQ) and completion queues (CQ) in GPU memory, as well as the technical means by which the GPU directly constructs and parses NVMe commands in these queues.

[0167] The RKey-based GPU memory direct write mechanism: This mechanism utilizes the RDMA memory registration mechanism to generate an RKey for the GPU memory and embeds it into NVMe commands, enabling remote target devices to directly write data to the GPU memory via RDMA Write.

[0168] The collaborative communication process between CPU and GPU: protecting the complete control flow interaction method across heterogeneous processors, from "GPU doorbell notification" to "CPU acquiring and packaging commands" and then to "CPU notifying GPU to complete".

[0169] Complete system implementation: Protects the overall device configuration of a hybrid computing system that includes all the above features.

[0170] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0171] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for NVMeoF remote storage based on direct GPU command control, characterized in that, The method is applied to an NVMe-oF remote storage system comprising an initiating host system and a remote target system. The initiating host system includes a GPU domain, a CPU domain, and a first RDMA network card. The remote target system includes a local NVMe SSD, NVMe-oF target software, and a second RDMA network card. The method includes the following steps: The CPU domain registers the data buffer in the GPU memory through the RDMA driver, obtains the remote access key RKey corresponding to the data buffer, and provides the remote access key RKey and the data buffer in the GPU memory to the GPU domain for constructing NVMe commands. The GPU domain and the CPU domain exchange information through a shared command descriptor located in the GPU memory. The shared command descriptor includes, but is not limited to, the address of the NVMe command in the GPU memory, the remote access key RKey of the GPU data buffer, and the address of the data buffer in the GPU memory. The application in the GPU domain initiates a read request, generates a corresponding NVMe command in the GPU memory through the GPU computing core, and adds it to the commit queue SQ. The data pointer field DPTR of the NVMe command is the address of the data buffer in the GPU memory and the remote access key RKey corresponding to the data buffer. Then, the CPU domain is notified through a doorbell. The CPU domain encapsulates the NVMe commands in the submission queue SQ into NVMe over RDMA command frames and submits an RDMA Send request to the first RDMA network card. The first RDMA network card packages the corresponding NVMe over RDMA command frame into a network data packet according to the RDMA Send request and sends it to the second RDMA network card; The second RDMA network card receives network data packets. The NVMe-oF target software parses the NVMe overRDMA command frame from the network data packets to obtain the corresponding NVMe command, and submits it to the local NVMe SSD to perform a read operation to obtain the data in the local NVMe SSD. The NVMe-oF target software uses RDMA Write operation on the second RDMA network card and the first RDMA network card according to the information of the NVMe over RDMA command frame to send the data in the local NVMe SSD to the GPU memory. After the data is sent, a completion message is sent to the CPU domain. The CPU domain generates a completion event and adds it to the completion queue (CQ) in the GPU memory. The GPU domain polls the completion queue (CQ) through the GPU computing core. When the completion event is detected, the read operation is confirmed as successful, and the application is notified.

2. The NVMeoF remote storage method based on GPU direct command control according to claim 1, characterized in that, When the CPU domain is notified via the doorbell, the GPU computing core performs a write operation on the doorbell register to trigger a CPU interrupt, causing the CPU domain to read NVMe commands from the GPU memory commit queue (SQ) after the interrupt.

3. The NVMeoF remote storage method based on GPU direct command control according to claim 2, characterized in that, The doorbell register is an MMIO register mapped to the CPU address space.

4. The NVMeoF remote storage method based on GPU direct command control according to claim 1, characterized in that, When the GPU computing core generates the corresponding NVMe command in the GPU memory and adds it to the commit queue (SQ), it includes: A corresponding NVMe command is constructed at the tail of the commit queue SQ in the GPU memory. The opcode field OPC of the NVMe command is set to read operation, the start logical block address field SLBA is set to the start logical block address corresponding to the read request, the block number field NLB is set to the length of the data requested to be transferred in the read request, and the data pointer field DPTR is set to the address of the data buffer in the GPU memory and the remote access key RKey corresponding to the data buffer. After the fields of the NVMe command are filled, the tail pointer of the commit queue SQ is updated.

5. The NVMeoF remote storage method based on GPU direct command control according to claim 4, characterized in that, When the NVMe-oF target software performs RDMA Write operations on the second RDMA network card and the first RDMA network card based on the information in the NVMe over RDMA command frame, the specific steps include: The NVMe-oF target software sends the address of the data buffer in the data pointer field DPTR and the remote access key RKey to the second RDMA network card; The second RDMA network card directly writes the data in the local NVMe SSD to the first RDMA network card through RDMA Write, based on the address of the data buffer and the remote access key RKey. The first RDMA network card directly DMA-transfers data from the local NVMe SSD to the data buffer of the GPU's video memory, allowing data transfer in the local NVMe SSD to completely bypass the CPU and system memory at the initiating end.

6. The NVMeoF remote storage method based on GPU direct command control according to claim 1, characterized in that, When the GPU domain polls the completion queue CQ through the GPU computing core, the process includes: the GPU computing core periodically checking whether the head pointer and tail pointer of the completion queue CQ are equal; when it detects that the head pointer and tail pointer are not equal, it reads the completion event from the completion queue CQ, updates the head pointer of the completion queue CQ, and notifies the application that the read operation is complete.

7. The NVMeoF remote storage method based on GPU direct command control according to claim 1, characterized in that, The method further includes a step of performing a write operation on the remote target, specifically including: The application in the GPU domain initiates a write request, generates a corresponding NVMe write command in the GPU memory through the GPU computing core, and adds it to the commit queue SQ. The data pointer field of the NVMe write command includes the GPU memory address of the data to be written and the remote access key RKey. Then, the CPU domain is notified through a doorbell. The CPU domain encapsulates the NVMe write command in the submission queue SQ into an NVMe over RDMA command frame and submits an RDMA Send request to the first RDMA network card. The first RDMA network card packages the corresponding NVMe over RDMA command frame into a network data packet according to the RDMA Send request and sends it to the second RDMA network card; The second RDMA network card receives network data packets. The NVMe-oF target software parses the NVMe overRDMA command frame from the network data packets to obtain the corresponding NVMe write command and submits it to the local NVMe SSD. The NVMe-oF target software uses RDMA Read operation to read the data to be written from the GPU memory of the initiating end according to the GPU memory address and remote access key RKey, and writes the data to be written to the local NVMe SSD. After the writing is completed, the completion message is sent to the CPU domain through the second RDMA network card and the first RDMA network card in sequence. The CPU domain generates a completion event and adds it to the completion queue (CQ) in the GPU memory. The GPU domain polls the completion queue (CQ) through the GPU computing core. When the completion event is detected, the write operation is confirmed as successful, and the application is notified.

8. An NVMeoF remote storage system based on GPU direct command control, characterized in that, The system includes an initiating host system and a remote target system. The initiating host system includes a GPU domain, a CPU domain, and a first RDMA network card. The remote target system includes a local NVMe SSD, NVMe-oF target software, and a second RDMA network card. The NVMeoF remote storage system is programmed or configured to perform the steps of the NVMeoF remote storage method based on GPU direct command control as described in any one of claims 1 to 7.

Images