RPC communication system and method based on SSD and CPU shared memory

By adopting an integrated computing and storage hardware design between the SSD and the host CPU, the host can directly access the SSD memory and use the RPC protocol stack for data interaction. This solves the problems of low data throughput efficiency and real-time monitoring of traditional NVMe SSDs, and improves system performance and resource utilization efficiency.

CN118981387APending Publication Date: 2024-11-19SHANGHAI FENGXING ZHIYUAN TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411118897.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

When a traditional CPU accesses an NVMe SSD, the data interaction is entirely asynchronous, resulting in low data throughput efficiency, high resource overhead, and an inability to monitor the SSD's execution status in real time.

Method used

By using SSD hardware that integrates computing and storage, the host CPU directly accesses the SSD's memory address space and interacts with data through the RPC protocol stack. By utilizing the SSD's internal memory, data movement is reduced, enabling real-time monitoring.

Benefits of technology

It improves data throughput, reduces host resource overhead and data access latency, and enables real-time monitoring of SSD performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981387A_ABST
    Figure CN118981387A_ABST
Patent Text Reader

Abstract

The invention discloses an RPC (Remote Procedure Call) communication system and method based on an SSD (Solid State Disk) and CPU (Central Processing Unit) shared memory, in the system, a host is in communication connection with the SSD, the SSD adopts computable and storable hardware, and the SSD comprises a computing CPU and a storage CPU; the computing CPU performs bidirectional communication with the host based on an RPC protocol stack, the SSD comprises a memory, and the SSD shares a memory address space corresponding to the memory with the host in a CMB mode, so that the host and the computing CPU can perform direct real-time data interaction based on the memory address space. According to the technical scheme, the host can directly access and operate the specified memory address space in the SSD, the memory space in the SSD is fully utilized, data carrying operation between the host and a traditional SSD is avoided, resource overhead and data access delay of the host are reduced, the data throughput capacity is improved, and the execution state of the SSD can be monitored in real time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of network protocol technology, and in particular to an RPC communication system based on SSD and CPU shared memory and an RPC communication method based on SSD and CPU shared memory. Background Art

[0002] Currently, traditional CPUs can only access NVMe SSDs by issuing stored NVMe instructions through the PCIe bus. The host sets the Submission Queue and Complete Queue addresses in the memory and sends instructions to the SSD to complete data throughput. To understand the status of the hard disk, data reading and writing interaction with the hard disk is fully asynchronous. After sending a command, the CPU must wait for the Complete Queue feedback before it can retrieve the relevant data from the returned memory.

[0003] The NVMe protocol supports a new CMB feature in versions after the Linux 5.10 kernel. This feature allows SSDs to configure registers. When the host enumerates PCIe devices, the SSD can have another PCIe base address space (BAR) in addition to its own basic storage PCIe base address, claiming that this space is a memory space that the host can directly access. Registers are used to indicate whether it supports various CMB modes. For example, SQ and CQ can be enabled on the DDR memory space of the SSD module instead of the host memory. The host CPU has the authority to directly access the memory space (Client memory Buffer) specified by the SSD. Summary of the invention

[0004] In view of the above problems, the present invention provides an RPC communication system and method based on SSD and CPU shared memory. By adopting computing and storage integrated hardware as SSD connected to the host CPU or host GPU, the host can directly access and operate the specified memory address space in the SSD, make full use of the memory space inside the SSD, avoid data transfer operations between the host and the traditional SSD, reduce the host's resource overhead and data access delay, accelerate data throughput, and can monitor the execution status of the SSD in real time.

[0005] To achieve the above object, the present invention provides an RPC communication system based on SSD and CPU shared memory, wherein the host is communicatively connected with the SSD, the SSD adopts computable storage hardware, and the SSD includes a computing CPU and a storage CPU;

[0006] The computing CPU performs bidirectional communication with the host based on the RPC protocol stack. The SSD includes a memory. The SSD shares a memory address space corresponding to the memory with the host in CMB mode, so that the host and the computing CPU can perform direct real-time data interaction based on the memory address space.

[0007] In the above technical solution, preferably, the multiple hosts are connected via RDMA / RoCE protocol or network card communication, and the memory address spaces corresponding to the hosts share memory via RPC protocol stack;

[0008] When the computing or storage resources of one of the hosts are limited, the remaining idle hosts are notified through RPC network protocol communication, and the data to be analyzed or stored is directly transferred across hosts to the memory address space of the idle hosts for computing or storage operations.

[0009] In the above technical solution, preferably, the computing CPU interacts with the host's management program through messages to achieve real-time response of all out-of-band management protocols, and the computing CPU interacts with the host's operation and maintenance control software platform through data to achieve direct presentation of supervision data without forwarding.

[0010] In the above technical solution, preferably, the host and the SSD perform message data interaction, analysis data interaction and analysis result interaction based on the RPC protocol stack;

[0011] The message data includes the system status of the SSD, the computing control command sent by the SSD in the host chassis, and the analysis and execution status of the data by the SSD;

[0012] The analysis data includes the to-be-processed data filled in by the host in the memory address space, and the data corresponding to the NVMe user instruction sent by the host or the RPC message sniffed in the memory address space by the SSD;

[0013] The analysis result includes the data result of the analysis completed by the SSD in the memory address space.

[0014] The present invention further proposes an RPC communication method based on SSD and CPU shared memory, which is applied to the RPC communication system based on SSD and CPU shared memory disclosed in any one of the above technical solutions, including:

[0015] The host writes the cached data to the memory address space of the memory in the SSD connected to the communication;

[0016] In CMB mode, the host sends an operation instruction for the data to the SSD;

[0017] The SSD directly performs corresponding operations on the data in the memory address space according to the operation instruction by the computing CPU;

[0018] After the operation is completed, an operation completion message is returned to the host.

[0019] In the above technical solution, preferably, the RPC communication method based on SSD and CPU shared memory also includes:

[0020] When the computing or storage resources of one of the multiple communication-connected hosts are limited, the remaining idle hosts are notified through the RPC protocol, and the data to be analyzed or stored is directly transferred across hosts to the memory address space of the idle host through the RPC protocol for computing or storage operations.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: by using computing and storage integrated hardware as an SSD connected to a host CPU or a host GPU, the host can directly access and operate the memory address space specified in the SSD, making full use of the memory space inside the SSD, avoiding data transfer operations between the host and the traditional SSD, reducing the host's resource overhead and data access delay, accelerating data throughput, and being able to monitor the execution status of the SSD in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the architecture of an RPC communication system based on SSD and CPU shared memory disclosed in an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of a specific structure of an SSD disclosed in an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of two-way communication interaction between a client and a server disclosed in an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of bidirectional communication interaction between a host and an SSD based on shared memory disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0028] like Figure 1 and Figure 2 As shown, according to an RPC communication system based on SSD and CPU shared memory provided by the present invention, the host is connected to the SSD for communication, the SSD adopts computable storage hardware, and the SSD includes a computing CPU and a storage CPU;

[0029] The computing CPU communicates bidirectionally with the host based on the RPC protocol stack. The SSD includes memory. The SSD shares the memory address space corresponding to the memory with the host in CMB mode, so that the host and the computing CPU can directly interact with real-time data based on the memory address space.

[0030] In this embodiment, by using integrated computing and storage hardware as an SSD connected to the host CPU or host GPU, the host can directly access and operate the specified memory address space in the SSD, fully utilize the memory space inside the SSD, avoid data transfer operations between the host and the traditional SSD, reduce the host's resource overhead and data access delay, accelerate data throughput, and be able to monitor the execution status of the SSD in real time.

[0031] Specifically, the SSD controller is allowed to map the general buffer in the SSD to the host side, and the host side can directly access the general buffer in the SSD through PCIe memory read / write. This has the following benefits:

[0032] 1) Make full use of the RAM storage space inside the device;

[0033] 2) In some cases, the number of PCIe DMA operations performed by the host is saved. For example, if NVMe SQ is established on CMB, the host only needs to write SQE (Submission Queue Entry) once, and the device no longer needs to pass through the PCIe bus when obtaining SQE.

[0034] 3) The general buffer inside the SSD can be used as a buffer from the network card device to the NVMe storage device (in the NVM-oF scenario), reducing CPU overhead and data access latency through PCIe point-to-point technology.

[0035] like Figure 3 As shown in FIG. 1 , by running an operating system in the CPU, DDR memory and storage inside the computational storage, the Remote Procedure Call Protocol (RPC) protocol stack is implemented, which can directly perform two-way communication interaction between the client and the server.

[0036] In the above implementation, preferably, the computing CPU interacts with the host's management program through messages to achieve real-time response of all out-of-band management protocols, and the computing CPU interacts with the host's operation and maintenance control software platform through data to achieve direct presentation of supervision data without forwarding.

[0037] The host and SSD are connected through the RPC protocol, which replaces the traditional expensive out-of-band management capabilities of SSD, NVMeManagement Interface. In this mode, the I2C / SMBUS interface is required to be combined with an external MCU to realize the monitoring part including the temperature, power consumption, and operating status (whether it is Panic, etc.) of the disk; management includes obtaining basic information of the disk (such as MN, PCIe ID, manufacturer name, Form Factor), upgrading firmware, and performing Format operations on the device. Another way is to obtain relevant protocols and business processing from the hard disk itself through the standard NVMe protocol.

[0038] The internal storage area of ​​SSD, i.e. computational storage, includes storage function area and computing function area. The actual storage function area inside the computational storage runs independently. The computing CPU in the computing function area inside the computational storage runs the RPC protocol. Combined with the shared memory in CMB mode, it can interact with the host management program in real time and realize the real-time response of all out-of-band management protocols. Even if the NVMe system fails, the computing function area can still output all the status of the NVMe system in real time.

[0039] SSD can be connected to the RPC-based operation and maintenance control software platform on the host side to directly display data, avoiding the problem of out-of-band management, NVME, etc., which requires data to be transferred to the supervision platform (such as Prometheus, Grafanar, ELK and other frameworks) through RPC. In addition, this interface has almost no impact on the actual SSD data throughput performance, because the CPU directly accesses the SSD's memory area, and only the DMA data transfer interaction on the physical PCIe link. Through RPC interaction, in addition to the standard NVMe protocol for NVMe computational storage, the SSD can perform calculations and feedback results through a message mechanism.

[0040] In the above implementation, preferably, the host and the SSD perform message data interaction, analysis data interaction and analysis result interaction based on the RPC protocol stack;

[0041] The message data includes the system status of the SSD, the computing control commands sent by the host chassis SSD, and the analysis and execution status of the data by the SSD;

[0042] The analyzed data includes the data to be processed filled in the memory address space by the host, and the data corresponding to the NVMe user instructions sent by the host or the RPC message sniffed in the memory address space by the SSD;

[0043] The analysis result includes the data result of the analysis completed by the SSD in the memory address space.

[0044] Specifically, the following is the specific process of implementing three types of data interaction through shared memory:

[0045] (1) Message data interaction

[0046] The host can obtain various system states of the SSD and can send computing control commands through the protocol to be accepted by the SSD. The actual analysis is executed on the SSD, and the host CPU can monitor the execution status in real time through RPC.

[0047] (2) Analyze data interaction

[0048] The host can fill in the data to be processed through the shared memory area, and the SSD can analyze the data according to the NVMe user instructions sent by the host or the RPC message sniffing of the shared memory area. The advantage is that the PCIe interface directly writes to the SSD memory, and the bandwidth and latency can be close to the speed of the host directly accessing the motherboard memory, which can replace the host's memory. At the same time, in this mode, the CPU can be replaced with the GPU when the GPU graphics card driver supports it, and when there is a DPU RDMA network card support, it can be replaced with a DPU network card, and both can access the SSD cache area to accelerate the throughput of the overall system.

[0049] (3) Analysis results interaction

[0050] The storage data and calculation data of computable storage are integrated. The analyzed data results can be filled in the shared memory area and the host can be directly notified for subsequent data calculation or read and write processing.

[0051] like Figure 4 As shown, according to the above implementation method, during the implementation process, a host is executing a database application, and a large amount of data written by the database software needs to be written to the cache first to increase the speed, and then needs to be actually written to the hard disk. The CMB mode can be directly used to write the data to the shared memory area supported by the SSD, and then an RPC message is sent to request the computing CPU of the SSD to notify the storage CPU to directly use the hardware in the chip to write the memory data directly to the hard disk, without the need to move it from the external memory to the hard disk, and the host CPU does not need to send additional instructions.

[0052] In this scenario, when the data capacity of cache and persistent write reaches a certain threshold, the database will require data merging and deduplication processing. The usual steps are to read data from the hard disk to the host memory, then use the host CPU to perform a round of merging and deduplication, and then send instructions to write back to the hard disk. In the CMB+RPC mode of the present invention, a message can be sent directly to the SSD to perform merging and deduplication and disk writing operations. At this time, all hard disk data does not need to be moved to the host memory, and can be directly merged and deduplicated in the hard disk and written back to the hard disk. The updated file system information can be returned directly to the host through RPC to update the corresponding file table and file system information of the host. It is equivalent to only sending a message to the hard disk, without moving the data. The hard disk completes the entire read and write processing and update process of the file system, and returns an updated message to the host to complete all processing, saving the overhead of the database corresponding processing that originally required frequent data transfer between the SSD and the host memory.

[0053] In the above implementation, preferably, multiple hosts are connected via RDMA / RoCE protocol or network card communication, and memory address spaces corresponding to the hosts share memory via RPC protocol stack;

[0054] When the computing or storage resources of one of the hosts are limited, the remaining idle hosts are notified through RPC network protocol communication, and the data to be analyzed or stored is directly transferred across hosts to the memory address space of the idle hosts for computing or storage operations.

[0055] Specifically, this model can be extended to storage sharing and message communication between multiple hosts. Because RPC is a network protocol, it can be done through standard Socket, and the current RDMA / RoCE and network cards can support the connection of network storage file system (NFS) or distributed file system (such as Ceph, Lustre, etc.). When a host finds that its computing or storage resources are limited, it can notify another idle host through network RPC messages that it needs to cooperate in processing part of the database or storage call processing business.

[0056] The actual application of idle hosts can be divided into two types:

[0057] (1) If there is an RDMA card or RoCE protocol, the network protocol can directly fill in data or messages to the shared memory area (CMB) mode of the storage and computing module and take over directly.

[0058] (2) If only the network cards are interconnected, the RPC message is first sent to the CPU of the idle host. After receiving the instruction, the CPU of the idle host can notify the storage and computing hardware of the local machine through another RPC address channel to complete the subsequent processing work.

[0059] Whether in RDMA / RoCE or pure NIC mode, the data required for cross-machine data analysis is directly moved by the cross-machine DPU / NIC / PCIe to network hardware, because RDMA can directly move data across machines to the shared storage area (CMB) of the SSD. The NIC can send data packets to the specified memory space through DPDK (it can also be opened in the CMB area of ​​the hard disk)

[0060] The present invention further proposes an RPC communication method based on SSD and CPU shared memory, which is applied to the RPC communication system based on SSD and CPU shared memory disclosed in any one of the above embodiments, including:

[0061] The host writes the cached data to the memory address space of the memory in the SSD connected to the communication;

[0062] In CMB mode, the host sends data-specific operation instructions to the SSD;

[0063] SSD directly performs corresponding operations on the data in the memory address space according to the operation instructions by the computing CPU;

[0064] After the operation is completed, an operation completion message is returned to the host.

[0065] In the above implementation, preferably, the RPC communication method based on SSD and CPU shared memory also includes:

[0066] When the computing or storage resources of one of the multiple communication-connected hosts are limited, the remaining idle hosts are notified through the RPC protocol, and the data to be analyzed or stored is directly transferred across hosts to the memory address space of the idle host through the RPC protocol for computing or storage operations.

[0067] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An RPC communication system based on SSD and CPU shared memory, characterized in that: The host is in communication connection with the SSD, the SSD adopts computable storage hardware, and the SSD includes a computing CPU and a storage CPU; The computing CPU performs bidirectional communication with the host based on the RPC protocol stack. The SSD includes a memory. The SSD shares a memory address space corresponding to the memory with the host in CMB mode, so that the host and the computing CPU can perform direct real-time data interaction based on the memory address space.

2. The RPC communication system based on SSD and CPU shared memory according to claim 1, characterized in that: Also includes: The multiple hosts are connected via RDMA / RoCE protocol or network card communication, and the memory address spaces corresponding to the hosts share memory via RPC protocol stack; When the computing or storage resources of one of the hosts are limited, the remaining idle hosts are notified through RPC network protocol communication, and the data to be analyzed or stored is directly transferred across hosts to the memory address space of the idle hosts for computing or storage operations.

3. The RPC communication system based on SSD and CPU shared memory according to claim 1 or 2, characterized in that: The computing CPU exchanges messages with the host's management program to achieve real-time response to all out-of-band management protocols, and the computing CPU exchanges data with the host's operation and maintenance control software platform to achieve direct presentation of supervision data without forwarding.

4. The RPC communication system based on SSD and CPU shared memory according to claim 3, characterized in that: The host and the SSD perform message data interaction, analysis data interaction and analysis result interaction based on the RPC protocol stack; The message data includes the system status of the SSD, the computing control command sent by the SSD in the host chassis, and the analysis and execution status of the data by the SSD; The analysis data includes the to-be-processed data filled in by the host in the memory address space, and the data corresponding to the NVMe user instruction sent by the host or the RPC message sniffed in the memory address space by the SSD; The analysis result includes the data result of the analysis completed by the SSD in the memory address space.

5. An RPC communication method based on SSD and CPU shared memory, characterized in that: The RPC communication system based on SSD and CPU shared memory as claimed in any one of claims 1 to 4 comprises: The host writes the cached data to the memory address space of the memory in the SSD connected to the communication; In CMB mode, the host sends an operation instruction for the data to the SSD; The SSD directly performs corresponding operations on the data in the memory address space according to the operation instruction by the computing CPU; After the operation is completed, an operation completion message is returned to the host.

6. The RPC communication method based on SSD and CPU shared memory according to claim 5, characterized in that: Also includes: When the computing or storage resources of one of the multiple communication-connected hosts are limited, the remaining idle hosts are notified through the RPC protocol, and the data to be analyzed or stored is directly transferred across hosts to the memory address space of the idle hosts through the RPC protocol for computing or storage operations.

Citation Information

Patent Citations

  • Data storage and calculation integrated solid state disk module and system

    CN114647380A

  • Solid state disk, data storage method and electronic equipment

    CN116932423A

  • Distributed storage data transmission method, device, system, equipment and medium

    CN117573043A

  • Data access method and apparatus, and storage medium

    WO2022160308A1

Cited By

  • Distributed file system based on data processing unit (DPU)

    CN119597722A

  • Distributed file system based on data processing unit (DPU)

    CN119597722B

  • Method and system for optimizing persistent cache performance of Lustre client based on GPU direct access

    CN121364832A

  • GPU direct access-based lustre client persistent cache performance optimization method and system

    CN121364832B