Access system and access method for NVMeoF storage based on CXL

Abstract

The invention discloses an access system and an access method for NVMeoF storage based on a CXL. The access method comprises the steps that a Host end creates a virtual NVMe block device through an NVMeoF driver; the Host and the Target are respectively connected with a CXL SWICTH end, and carry out data transmission through a CXL.mem protocol; a physical memory area shared by the Host end and the Target end is configured in the CXL SWICTH end, and an SQ queue and a CQ queue which are respectively created by driving programs of the Host end and the Target end are arranged in the shared physical memory area; and the Target end is used for acquiring the SQE from the shared memory and finally converting the SQE into an operation on the local NVMe, and writing the CQE into the CQ of the shared memory. And the Host end is used for generating the SQE, putting the SQE into the corresponding SQ and processing the CQE in the CQ. According to the application, the cost is low, and relatively high performance is obtained.

Description

Technical Field

[0001] The present application relates to the technical field of NVMeoF storage access, and in particular to an NVMeoF storage access system and access method. Background Art

[0002] The current NVMeoF technology can be roughly divided into three categories according to the transmission protocol: NVMe over FC (SAN network based on fiber), NVMe over TCP (based on traditional Ethernet), and NVMe over RDMA (based on RDMA protocol). However, although NVMe over FC and NVMe over RDMA provide high-performance data transmission, NVMe over FC relies on Fiber Channel technology, and NVMe over RDMA relies on network technologies that support RDMA, such as InfiniBand or RoCE. These technologies enable servers to efficiently access storage devices remotely, but usually require investment in expensive dedicated hardware. This limits its widespread application. Although NVMe over TCP has a lower cost, its performance is relatively poor. Summary of the invention

[0003] Based on this, in order to solve the above technical problems, a CXL-based NVMeoF storage access system and access method are provided to solve the problem that the existing technology cannot take into account both low cost and high performance.

[0004] In a first aspect, a CXL-based NVMeoF storage access system, the NVMeoF storage system comprising: a host end and a target end both supporting CXL and NVMeoF technologies, and a CXL SWICTH end;

[0005] The host device creates a corresponding virtual NVMe block device through the NVMeoF driver; the host and target are respectively connected to the CXL SWICTH end, and data transmission is performed through the CXL.mem protocol; a physical memory area that can be shared by the host and target is configured in the CXLSWICTH end, so that both the host and target can access the shared physical memory area;

[0006] In the shared physical memory area of ​​the CXL SWICTH end, the SQ and CQ queues of the Host end created by the Host end driver and the SQ and CQ queues of NVMe created by the Target end driver are set;

[0007] The Target end is used to monitor the SQ queue of the Host end in real time, and write the unprocessed SQE in the SQ queue of the Host end into the SQ queue of the NVMe end; it is also used to generate CQE after executing the SQE in the SQ queue of NVMe, and put it into the CQ queue of NVMe; it is also used to write the CQE in the CQ queue of NVMe into the CQ queue of the Host end;

[0008] The Host end is used to initiate an IO operation on NVMe, and encapsulate the IO operation into an SQE and put it into the SQ queue of the Host end; it is also used to obtain the CQE from the CQ queue of the Host end.

[0009] In the above solution, optionally, the Target end is connected to the NVMe device via the PCIe protocol.

[0010] In the above solution, optionally, the Host end and the Target end are connected to the CXL SWICTH via a PCIe interface.

[0011] In the above scheme, optionally, the shared physical memory area of ​​the CXL SWICTH end device is also provided with an Info area, and the Info area is used to record metadata, and the metadata includes: the status of the Host end, the status of the Target end, the current number of SQs and CQs, the offset of SQs and CQs relative to the starting position of the shared memory, and the head and tail pointer positions of SQs and CQs.

[0012] In the above scheme, optionally, the Host-side driver determines the number of SQ and CQ queues to be created on the Host side according to the number of cores of the Host-side CPU, and the Target-side driver determines the number of SQ and CQ queues to be created for NVMe according to the number of cores of the Target-side CPU.

[0013] In a second aspect, a method for accessing a CXL-based NVMeoF storage access system is provided, wherein the method adopts the CXL-based NVMeoF storage access system according to any one of the first aspects, and the method includes:

[0014] Initiate polling to the physical memory area shared by the Host and Target in the CXL SWICTH end. If it is detected that the SQ queue of the Host end is updated and there are unprocessed SQEs, convert the unprocessed SQEs in the SQ queue of the Host end into the NVMe corresponding format and write them into the SQ queue of the NVMe end; the unprocessed SQE is used by the Host end to initiate an IO operation on NVMe, encapsulate the IO operation into an SQE and put it into the SQ queue of the Host end;

[0015] Get the unprocessed SQE from the SQ queue on the NVMe side and execute it, generate CQE, and write it to the CQ queue on the NVMe side;

[0016] Initiate polling to the physical memory area shared by the Host and Target in the CXL SWICTH end. If it is detected that the CQ queue on the NVMe end is updated and there are unprocessed CQEs, convert the unprocessed CQEs in the CQ queue on the NVMe end into the format corresponding to the Host and write them into the CQ queue on the Host end to wait for the Host end to obtain them.

[0017] This application has at least the following beneficial effects:

[0018] This application is connected to the CXL SWICTH end through the Host end and the Target end respectively, and data transmission is carried out through the CXL.mem protocol to realize memory sharing, and a shared memory area for storing memory access data is constructed in the CXL SWICTH end, and SQ and CQ queues are constructed and managed in this area. Therefore, by using standard CXL interfaces and NVMe devices, dependence on expensive dedicated hardware is avoided. SQ and CQ queues are directly created and managed in shared memory, and data is transmitted through SQ and CQ queues, which reduces the transmission delay of data between the Host end and the Target end, thereby improving the data processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a structural diagram of NVMeoF in the prior art;

[0020] Figure 2 A structural diagram of a CXL-based NVMeoF storage access system provided in one embodiment of the present application;

[0021] Figure 3 A hardware structure diagram of a CXL-based NVMeoF storage access system provided in one embodiment of the present application;

[0022] Figure 4 A schematic diagram of a queue constructed by the Host in CXL SWICTH according to an embodiment of the present application;

[0023] Figure 5 A schematic diagram of a queue constructed in CXL SWICTH by a Target end provided in an embodiment of the present application;

[0024] Figure 6 The actual data distribution in the shared memory of the CXLSWICTH end of a CXL-based NVMeoF storage access system provided in one embodiment of the present application;

[0025] Figure 7 Four state diagrams of the Host and Target ends provided for one embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] like Figure 1 As shown, each structure in NVMeoF is shown. NVMeoF is divided into Host and Target, allowing Host and Target to be on different hosts. Host can communicate with Target through various protocols while maintaining ultra-low latency and high bandwidth. Currently, NVMe supports NVMeoF technology using RDMA, Fibre Channel (FC) and TCP. In order to solve the pain points of high cost and low performance in the current NVMeoF technology, this application proposes a CXL-based NVMeoF technology.

[0028] In one embodiment, Figure 2 As shown, a CXL-based NVMeoF storage access system is provided, and the NVMeoF storage system includes: a host end and a target end both supporting CXL and NVMeoF technologies, and a CXL SWICTH end;

[0029] The host device creates a corresponding virtual NVMe block device through the NVMeoF driver; the host and target are respectively connected to the CXL SWICTH end, and data transmission is performed through the CXL.mem protocol; a physical memory area that can be shared by the host and target is configured in the CXLSWICTH end, so that both the host and target can access the shared physical memory area;

[0030] In the shared physical memory area of ​​the CXL SWICTH end, the SQ and CQ queues of the Host end created by the Host end driver and the SQ and CQ queues of NVMe created by the Target end driver are set;

[0031] The Target end is used to monitor the SQ queue of the Host end in real time, and write the unprocessed SQE in the SQ queue of the Host end into the SQ queue of the NVMe end; it is also used to generate CQE after executing the SQE in the SQ queue of NVMe, and put it into the CQ queue of NVMe; it is also used to write the CQE in the CQ queue of NVMe into the CQ queue of the Host end;

[0032] The Host end is used to initiate an IO operation on NVMe, and encapsulate the IO operation into an SQE and put it into the SQ queue of the Host end; it is also used to obtain the CQE from the CQ queue of the Host end.

[0033] Specifically, the actual NVMe device on the Target side is connected to the Target through the PCIe protocol, and the Host side creates the corresponding virtual NVMe block device through the NVMeoF driver. The Host and Target are connected through CXL SWICTH, and memory sharing is achieved through the CXL.mem protocol. The Host driver and the Target driver build the NVMeoF protocol through shared memory. Through the NVMeoF over CXL protocol, the read and write of the virtual NVMe block device on the Host side will be mapped to the read and write of the actual NVMe block device on the Target side.

[0034] Among them, Figure 3 As shown in the figure, both the Host node and the Target node are connected to the SWITCH through the PCIe interface, and the NVMeSSD is connected to the Target node through PCIe. CXL SWITCH implements the shared memory function between the Host and Target hosts. The hardware functions are provided by the CXL SWITCH chip. By pointing the access of the Host and Target nodes to a certain address space to the same storage medium, memory sharing can be achieved on the Host and Target ends. By dividing the shared memory into different areas, a new communication protocol can be built on the shared memory.

[0035] The main task of the host driver is to generate the host SQE and put it into the corresponding SQ and process the CQE in the CQ. The main task of the target driver is to obtain the SQE from the shared memory and finally convert it into the operation of the local NVMe and write the CQE into the CQ of the shared memory.

[0036] like Figure 4 As shown in the figure, the kernel driver design is shown, which is the SQ and CQ queues created in the shared memory on the Host side. The Host and Target access the same memory address space through the shared memory. In this memory space, the Host side establishes multiple pairs of SQ and CQ according to the NVMe protocol. The Host side operates on SQ and CQ, and the Target side can obtain updated data in real time.

[0037] Figure 5The NVMe SQ / CQ queues on the Target side are shown. The interaction between the Target side and the NVMe controller also uses the SQ and CQ queues. The Target side takes out the SQE generated by the Host side and converts it into the SQE in NVMe format, then takes out the CQE from the NVMe CQ and converts it into the CQE in the Host format.

[0038] In the above-mentioned CXL-based NVMeoF storage access system, the Host and Target are respectively connected to the CXLSWICTH end, and data transmission is performed through the CXL.mem protocol to realize memory sharing, and a shared memory area for storing memory access data is constructed in the CXL SWICTH end, and SQ and CQ queues are constructed and managed in this area. Therefore, by using standard CXL interfaces and NVMe devices, dependence on expensive dedicated hardware is avoided. SQ and CQ queues are directly created and managed in shared memory, and data is transmitted through SQ and CQ queues, which reduces the transmission delay of data between the Host and Target ends, thereby improving the data processing speed.

[0039] In one embodiment, the shared physical memory area of ​​the CXL SWICTH end device is further provided with an Info area, and the Info area is used to record metadata, and the metadata includes: the status of the Host end, the status of the Target end, the current number of SQs and CQs, the offset of SQs and CQs relative to the starting position of the shared memory, and the head and tail pointer positions of SQs and CQs.

[0040] Specifically, Figure 6As shown, the actual data distribution of CXL-based NVMeof technology in shared memory is shown. It can be seen that the CXL shared memory is divided into two areas, the Info area and the Data area. The Info area is responsible for recording some metadata, the H_status field is responsible for recording the status of the Host, and the T_status field is responsible for recording the status of the Target. The QPNumber field is responsible for the current number of QPs, and the QP Off field is responsible for recording the offset of the QP in the Data area relative to the starting position of the shared memory. The QP Info area is responsible for recording the head and tail pointer positions of the SQ and CQ in each QP. The Data area is responsible for recording the SQE and CQE queues of each QP. The SQ is updated by the Host, and the CQ is updated by the Target. Every time the Host initiates IO, the SQE stored in the driver will be filled into the SQ of the corresponding QP. And update the SQ Tail of the corresponding QP in the Info area. The Target is responsible for polling the Host to see if there is any IO issued. When it detects that there is data update, the Target will fetch the corresponding SQE from the corresponding SQ. After processing, it will update the head of the corresponding SQ, generate the corresponding CQE, and fill it into the CQ of the corresponding QP, and update the tail of the corresponding CQ in the Info area. If the Host finds that the Target has processed the CQ through polling, it will fetch the CQE from the CQ for processing and update the head of the corresponding CQ. This IO is completed.

[0041] In one embodiment, the Host-side driver determines the number of SQ and CQ queues to be created on the Host-side according to the number of cores of the Host-side CPU, and the Target-side driver determines the number of SQ and CQ queues of NVMe to be created according to the number of cores of the Target-side CPU.

[0042] In one embodiment, a method for accessing a CXL-based NVMeoF storage access system is proposed. The method adopts the CXL-based NVMeoF storage access system described above and is applied to a Target end. The method includes:

[0043] Initiate polling to the physical memory area shared by the Host and Target in the CXL SWICTH end. If it is detected that the SQ queue of the Host end is updated and there are unprocessed SQEs, convert the unprocessed SQEs in the SQ queue of the Host end into the NVMe corresponding format and write them into the SQ queue of the NVMe end; the unprocessed SQE is used by the Host end to initiate an IO operation on NVMe, encapsulate the IO operation into an SQE and put it into the SQ queue of the Host end;

[0044] Get the unprocessed SQE from the SQ queue on the NVMe side and execute it, generate CQE, and write it to the CQ queue on the NVMe side;

[0045] Initiate polling to the physical memory area shared by the Host and Target in the CXL SWICTH end. If it is detected that the CQ queue on the NVMe end is updated and there are unprocessed CQEs, convert the unprocessed CQEs in the CQ queue on the NVMe end into the format corresponding to the Host and write them into the CQ queue on the Host end to wait for the Host end to obtain them.

[0046] Specifically, the software implementation steps of CXL-based NVMeoF technology include:

[0047] Initialization phase (configuration phase):

[0048] (1) The host creates NVMeoF SQ and CQ queues in the CXL shared memory according to the number of CPU cores.

[0049] (2) The Target initializes NVMe and creates the corresponding working thread.

[0050] Processing stage:

[0051] (1) The host initiates an IO operation on NVMe. The host driver is responsible for encapsulating the read IO into an SQE, placing it in the SQ queue created in the shared memory, and updating the SQ Tail.

[0052] (2) When the thread driven by the Target side takes turns to update the corresponding SQ, it will immediately take out the SQE from the SQ, convert it into the SQE in the corresponding format of NVMe, and write it to the SQ queue of NVMe.

[0053] (3) Target-side NVMe processes the SQE, generates a CQE, and writes it into the CQ.

[0054] (4) The thread driven by the Target side takes turns to update the CQ on the NVMe side, immediately takes out the CQE from the CQ, converts it into the format of the Host CQE, and writes it to the CQ on the Host side.

[0055] (5) The host driver polls the CQ for updates, immediately retrieves the corresponding CQE from the CQ, checks the result of the CQE, and completes the IO.

[0056] like Figure 7As shown in the figure, the state transition of Host and Target is shown. Host and Target maintain four states respectively, namely Reset, Ready, Connected and Disconnected. After the driver is loaded, the state of Host and Target will be set to Reset by default. After the Target is ready to be connected, it will change its state to Ready. When the Host tries to connect to the Target using the nvme tool, if it finds that the Target is already in Ready state, it will first set itself to Connected state, indicating that the Host connection has been established. At this time, the Target will also change its status to Connected state, and the connection is successfully established. After the connection is completed, the Host fills the Info area according to the number of QPs created, and updates its own status to Ready state, and the Target also updates its own status to Ready. When the host and target are successfully connected (Host polls the Target's status, if the Target's status becomes Ready, the Host changes its own status to Connected, and the Target polls the Host's status, if the Host's status becomes Connected, the Target changes its own status to Connected), it indicates that the connection has been completed. If the Host tries to end the connection, the Host first processes all CQEs in the CQ, and then updates its own status to Disconnected. The Target then finds that the Host's status has been set to Disconnected, processes all SQEs in the SQ, updates the results to CQEs in the CQ, and then sets its own status to Disconnected. Next, the Host finds that the Target's status has been Disconnected, and needs to query the CQ again to see if there are any CQEs that need to be processed. If so, after processing all CQEs, it updates its own status to Ready.

[0057] The software consists of two parts: user-mode tools and kernel-mode drivers. The kernel-mode drivers are divided into host-side drivers and target-side drivers. User-mode tools are mainly used to implement basic functions such as discovery, connection and disconnection. The command format is as follows:

[0058] Discover: nvme-discover -t cxl -a $CXL_ID;

[0059] Connect: nvme connect -t cxl -a $CXL_ID;

[0060] Disconnect: nvme disconnect -n $NAME;

[0061] CXL_ID is the CXL shared memory identity of the target, which is generated during initialization and is used to uniquely identify the peer. NAME is the name of the target's namespace, which can be obtained during the discover phase.

[0062] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0063] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A CXL-based NVMeoF storage access system, characterized in that: The NVMeoF storage system includes: a host end and a target end both supporting CXL and NVMeoF technologies, and a CXL SWICTH end; The host device creates a corresponding virtual NVMe block device through the NVMeoF driver; the host and target are respectively connected to the CXL SWICTH end, and data transmission is implemented through the CXL.mem protocol; a physical memory area that can be shared by the host and target is configured in the CXL SWICTH end, so that both the host and target can access the shared physical memory area; In the shared physical memory area of ​​the CXL SWICTH end, the SQ and CQ queues of the Host end created by the Host end driver and the SQ and CQ queues of NVMe created by the Target end driver are set; The Target end is used to monitor the SQ queue of the Host end in real time, and write the unprocessed SQE in the SQ queue of the Host end into the SQ queue of the NVMe end; it is also used to generate CQE after executing the SQE in the SQ queue of NVMe, and put it into the CQ queue of NVMe; it is also used to write the CQE in the CQ queue of NVMe into the CQ queue of the Host end; The Host end is used to initiate an IO operation on NVMe, and encapsulate the IO operation into an SQE and put it into the SQ queue of the Host end; it is also used to obtain the CQE from the CQ queue of the Host end.

2. The CXL-based NVMeoF storage access system according to claim 1, characterized in that: The Target end is connected to the NVMe device via the PCIe protocol.

3. The CXL-based NVMeoF storage access system according to claim 1, characterized in that: The Host end and the Target end are connected to the CXL SWICTH via a PCIe interface.

4. According to the CXL-based NVMeoF storage access system of claim 1, the shared physical memory area of ​​the CXL SWICTH end device is further provided with an Info area, the Info area is used to record metadata, and the metadata includes: The status of the Host, the status of the Target, the current number of SQs and CQs, the offset of SQs and CQs relative to the start position of the shared memory, and the positions of the head and tail pointers of SQs and CQs.

5. The CXL-based NVMeoF storage access system according to claim 1, characterized in that: The host-side driver determines the number of SQ and CQ queues to be created on the host side according to the number of cores of the host-side CPU, and the target-side driver determines the number of SQ and CQ queues of NVMe to be created according to the number of cores of the target-side CPU.

6. A method for accessing a CXL-based NVMeoF storage access system, characterized in that: The method adopts the CXL-based NVMeoF storage access system according to any one of claims 1 to 5, and is applied to a Target end. The method includes: Initiate polling to the physical memory area shared by the Host and Target in the CXL SWICTH end. If it is detected that the SQ queue of the Host end is updated and there are unprocessed SQEs, convert the unprocessed SQEs in the SQ queue of the Host end into the NVMe corresponding format and write them into the SQ queue of the NVMe end; the unprocessed SQE is used by the Host end to initiate an IO operation on NVMe, encapsulate the IO operation into an SQE and put it into the SQ queue of the Host end; Get the unprocessed SQE from the SQ queue on the NVMe side and execute it, generate CQE, and write it to the CQ queue on the NVMe side; Initiate polling to the physical memory area shared by the Host and Target in the CXL SWICTH end. If it is detected that the CQ queue on the NVMe end is updated and there are unprocessed CQEs, convert the unprocessed CQEs in the CQ queue on the NVMe end into the format corresponding to the Host and write them into the CQ queue on the Host end to wait for the Host end to obtain them.

Images