Multi-channel NVMe SSD storage acceleration system based on FPGA

Abstract

The invention discloses a multi-channel NVMe SSD (Non-Volatile Memory Express Solid State Disk) storage acceleration system based on an FPGA (Field Programmable Gate Array), which is used for reading and writing the multi-channel NVMe SSD and comprises an NVMe Target controller, a Raid controller, a DDR (Double Data Rate) controller and NVMe Host controllers, the number of the NVMe Host controllers corresponds to that of the NVMe SSD, and the NVMe Host controllers are connected with the NVMe SSD in a one-to-one correspondence manner. According to the FPGA-based multi-channel NVMe SSD storage acceleration system, the device end is presented to the CPU host end, so that the CPU host end performs synchronous read-write operation on the multi-channel NVMe SSD of the device end, storage resource pooling is realized, the CPU host end can conveniently manage and maintain the device end, and the working efficiency is further improved; in the system, the number of the NVMe SSDs depends on the number of links between the NVMe Host controller and the NVMe SSDs in the system, a PCIe hard core controller and a PCIe exchange chip are not needed, the whole system is simpler, and the cost is lower; and meanwhile, the RAID function in the scheme does not depend on an external CPU and is autonomously realized by the storage acceleration system.

Description

Technical Field

[0001] The present invention belongs to the field of high-speed recording and storage technology, and specifically relates to a multi-channel NVMe SSD storage acceleration system based on FPGA. Background Art

[0002] NVMe SSDs are increasingly being used as storage media in high-speed recording and storage systems. Compared to mechanical hard drives and SATA SSDs, they significantly improve read and write performance while enhancing system stability. To expand capacity and increase speed, systems typically use PCIe switch chips to connect multiple NVMe SSDs. However, there are only a few domestic manufacturers capable of producing domestically produced PCIe switch chips, and their product stability is limited. Therefore, a new alternative to PCIe switching is urgently needed. Most researchers have turned their attention to FPGAs, as some high-end FPGAs integrate multiple PCIe controllers, each capable of connecting an NVMe SSD. Furthermore, it is possible to build a PCIe soft-core controller using the logic resources within the FPGA to compensate for the shortage of native PCIe controllers.

[0003] In the existing technology, the enumeration and configuration of each SSD are completed by the host side. When working, control commands need to be issued to each SSD, and the operating system and CPU implement the RAID function. The implementation of the RAID function is relatively complex and inefficient. In the existing technology, each SSD is visible to the host side, and each SSD is used independently. The host side needs to manage and maintain each electronic disk, and storage resources cannot be pooled. At the same time, in the existing technology, the number of SSDs is controlled by the PCIe hard-core controller inside the FPGA or by the PCIe switching chip. However, for the former, only FPGA devices with integrated PCIe hard-core controllers can be selected, which limits the selection, and the maximum number of connected SSDs depends on the number of PCIe hard-core controllers. For the latter, PCIE chips need to be added, which not only increases hardware costs but also has a significant impact on power consumption. Summary of the Invention

[0004] The purpose of the present invention is to solve the problems raised in the background technology and propose a multi-channel NVMe SSD storage acceleration system based on FPGA.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] The present invention proposes an FPGA-based multi-channel NVMe SSD storage acceleration system, which is used to read and write multi-channel NVMe SSDs. The storage acceleration system includes an NVMe Target controller, a Raid controller, a DDR controller, and an NVMe Host controller corresponding to the number of NVMe SSDs. The NVMe Host controller is connected to the NVMe SSDs in a one-to-one correspondence, the NVMe Target controller is connected to the CPU, and the DDR controller is connected to a memory, wherein:

[0007] When performing read / write operations on a multi-way NVMe SSD:

[0008] The CPU is used to create a read / write IO command and notify the NVMe Target controller through the doorbell that there is a command to be executed;

[0009] The NVMe Target controller is used to extract the read / write IO command from the CPU after receiving the doorbell, parse the read / write IO command, and send the parsed read / write IO command to the Raid controller;

[0010] The Raid controller is configured to, after receiving the parsed read / write IO command, convert the parsed read / write IO command and send the converted read / write IO command to the DDR controller and each NVMe Host controller respectively; and after receiving the first data sent by the DDR controller, divide the first data into a preset number of data blocks, and poll and send each data block to each NVMe Host controller; and after receiving the data blocks sent by each NVMe Host controller, merge each data block in the order sent by the Raid controller to obtain second data, and send the second data to the DDR controller;

[0011] The DDR controller is configured to receive the second data sent by the Raid controller and write the second data into the memory after receiving the converted read IO command, and to read the first data from the memory and send the first data to the Raid controller after receiving the converted write IO command;

[0012] Each of the NVMe Host controllers is used to read the data block from the corresponding NVMe SSD and send it to the Raid controller after receiving the converted read IO command, and is used to receive the data block sent by the Raid controller and write it to the corresponding NVMe SSD after receiving the converted write IO command.

[0013] Preferably, the Raid controller is used to convert the parsed read / write IO command after receiving it, and send the converted read / write IO command to the DDR controller and each NVMe Host controller respectively, and operate as follows:

[0014] After receiving the parsed read IO command, the Raid controller converts the parsed read IO command into a first DDR control command and a first SSD control command corresponding to the number of NVMe Host controllers, and sends the first DDR control command to the DDR controller, and sends each first SSD control command to the NVMe Host controller one by one, wherein the first DDR control command and each first SSD control command serve as the converted read IO commands from the Raid controller to the DDR controller and each NVMe Host controller respectively;

[0015] After receiving the parsed write IO command, the Raid controller converts the parsed write IO command into a second DDR control command and a second SSD control command corresponding to the number of NVMe Host controllers, and sends the second DDR control command to the DDR controller, and sends each second SSD control command to the NVMe Host controller one by one, wherein the second DDR control command and each second SSD control command are used as the converted write IO commands sent by the Raid controller to the DDR controller and each NVMe Host controller respectively.

[0016] Preferably, after receiving the first DDR control command, the DDR controller receives the second data sent by the Raid controller and writes it into the memory, and after receiving the second DDR control command, reads the first data from the memory and sends it to the Raid controller.

[0017] Preferably, after receiving the first SSD control command, each NVMe Host controller reads the data block from the corresponding NVMe SSD and sends it to the Raid controller, and is used to receive the data block sent by the Raid controller and write it to the corresponding NVMe SSD after receiving the second SSD control command.

[0018] Preferably, the size of the data block is the stripe size, the number of data blocks written or read by each NVMe SSD is consistent, and when each NVMe Host controller reads the data block, all NVMe Host controllers read the data block in the order in which the Raid controller polls and sends it.

[0019] Preferably, the parsed read IO command and write IO command each include an operation code, a read or write data amount, an LBA address, and a memory address.

[0020] Preferably, the first DDR control command includes a write operation code, a first data amount, and a memory address;

[0021] The second DDR control command includes a read operation code, a second data amount, and a memory address;

[0022] Each of the first SSD control commands includes a read operation code, a third data amount, an LBA address, and a memory address, wherein a sum of the third data amounts in all the first SSD control commands is equal to the first data amount;

[0023] Each of the second SSD control commands includes a write operation code, a fourth data amount, an LBA address, and a memory address, wherein a sum of the fourth data amounts in all the second SSD control commands is equal to the second data amount.

[0024] Preferably, the memory is the memory space of the CPU or the memory space of the storage acceleration system.

[0025] Preferably, the FPGA-based multi-channel NVMe SSD storage acceleration system further includes initialization before the read / write operation, and the initialization process is as follows:

[0026] First, a link is established between each NVMe Host controller and the corresponding NVMe SSD, so that each NVMe Host controller obtains the in-place status information and capacity information of the corresponding NVMe SSD;

[0027] Then, a link is established between the CPU and the NVMe Target controller, so that the CPU can obtain the in-place status information and capacity information of each NVMe SSD.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] This FPGA-based multi-channel NVMe SSD storage acceleration system presents the device side to the CPU host side, allowing the CPU host side to perform synchronous read and write operations on the multi-channel NVMe SSDs on the device side, realizing storage resource pooling, facilitating the CPU host side's management and maintenance of the device side, and thus improving work efficiency.

[0030] In this system, the number of NVMe SSDs depends on the number of links between the NVMe Host controller and the NVMe SSDs in the system. No PCIe hard-core controller or PCIe switch chip is required, making the overall solution simpler and less costly. At the same time, the RAID function in this solution does not rely on an external CPU and is independently implemented by the storage acceleration system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1This is a module block diagram of the FPGA-based multi-channel NVMe SSD storage acceleration system of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0033] In one embodiment, Figure 1 As shown, a multi-channel NVMe SSD storage acceleration system based on FPGA is provided. The storage acceleration system is used to read and write multi-channel NVMe SSDs (wherein the number of NVMe SSDs is not limited and depends on the number of high-speed transceivers in the storage acceleration system and conventional logic resources). The storage acceleration system (i.e., FPGA) includes an NVMeTarget controller, a Raid controller, and a DDR controller, as well as an NVMe Host controller corresponding to the number of NVMe SSDs. The NVMe Host controller is connected to the NVMe SSD in a one-to-one correspondence (and each NVMe Host controller is connected to the corresponding NVMe SSD via a PCIE link). The NVMe Target controller is connected to the CPU, and the DDR controller is connected to a memory, wherein:

[0034] When performing read / write operations on a multi-way NVMe SSD:

[0035] The CPU creates read / write IO commands and notifies the NVMe Target controller through the doorbell that there are commands to be executed;

[0036] The NVMe Target controller is used to extract the read / write IO command from the CPU after receiving the doorbell, parse the read / write IO command, and send the parsed read / write IO command to the Raid controller; wherein, the parsed read IO command and write IO command both include the operation code, the amount of read or write data (the amount of read and write data is the same), the LBA address and the memory address, wherein the LBA address is the address of each NVMe SSD.

[0037] The Raid controller is used to convert the parsed read / write IO commands after receiving them, and send the converted read / write IO commands to the DDR controller and each NVMe Host controller respectively, in the following way:

[0038] After receiving the parsed read IO command, the Raid controller converts the parsed read IO command into a first DDR control command and a first SSD control command corresponding to the number of NVMe Host controllers, and sends the first DDR control command to the DDR controller, and sends each first SSD control command to the NVMe Host controller in a one-to-one correspondence, wherein the first DDR control command and each first SSD control command serve as the converted read IO commands sent by the Raid controller to the DDR controller and each NVMe Host controller respectively; wherein each first SSD control command includes a read operation code, a third data amount, an LBA address, and a memory address, wherein the sum of the third data amounts in all first SSD control commands is equal to the first data amount;

[0039] After receiving the parsed write IO command, the Raid controller converts the parsed write IO command into a second DDR control command and a second SSD control command corresponding to the number of NVMe Host controllers, and sends the second DDR control command to the DDR controller, and sends each second SSD control command to the NVMe Host controller in a one-to-one correspondence, wherein the second DDR control command and each second SSD control command are used as the converted write IO commands sent by the Raid controller to the DDR controller and each NVMe Host controller respectively, wherein each second SSD control command includes a write operation code, a fourth data amount, an LBA address, and a memory address, wherein the sum of the fourth data amount in all second SSD control commands is equal to the second data amount;

[0040] At the same time, after receiving the first data sent by the DDR controller, the Raid controller divides the first data into a preset number of data blocks (the size of the data block is the stripe size, which is 4KB in this embodiment. The specific number of divided data blocks depends on the amount of data written, the number of NVMe SSDs and the stripe size), and polls and sends each data block to each NVMe Host controller (for example, the number of NVMe Host controllers is 4, and the four NVMe Host controllers and all data blocks are numbered, the first data block is sent to the first NVMe Host controller, the second data block is sent to the second NVMe Host controller, the third data block is sent to the third NVMe Host controller, the fourth data block is sent to the fourth NVMe Host controller, the fifth data block is sent to the first NVMe Host controller, and so on in this order, and the number of data blocks received by each NVMe Host controller is equal, and each NVMe Host controller writes a data block to the corresponding NVMe SSD); and after receiving each NVMe After the Host controller sends the data block, it merges the data blocks in the order sent by the Raid controller to obtain the second data (that is, the order of the data blocks in the merged second data is still arranged in the order of the original numbers), and sends the second data to the DDR controller.

[0041] The DDR controller is used to receive the second data sent by the Raid controller and write it to the memory after receiving the converted read IO command, and to read the first data from the memory and send it to the Raid controller after receiving the converted write IO command; that is: after receiving the first DDR control command, the DDR controller receives the second data sent by the Raid controller and writes it to the memory, and to read the first data from the memory and send it to the Raid controller after receiving the second DDR control command.

[0042] Each NVMe Host controller is used to read the data block from the corresponding NVMe SSD and send it to the Raid controller after receiving the converted read IO command, and to receive the data block sent by the Raid controller and write it to the corresponding NVMe SSD after receiving the converted write IO command; that is, after receiving the first SSD control command, each NVMe Host controller reads the data block from the corresponding NVMe SSD and sends it to the Raid controller, and after receiving the second SSD control command, receives the data block sent by the Raid controller and writes it to the corresponding NVMe SSD (the number of data blocks written or read by each NVMe SSD is the same, and when each NVMe Host controller reads the data block, all NVMe Host controllers read the data block in the order sent by the Raid controller in round-robin fashion, that is, the first NVMe Host controller reads the first data block from the corresponding NVMe SSD, the second NVMe Host controller reads the second data block from the corresponding NVMe SSD, the third NVMe Host controller reads the third data block from the corresponding NVMe SSD, the fourth NVMe Host controller reads the fourth data block from the corresponding NVMe SSD, and the first NVMe The host controller reads the fifth data block from the corresponding NVMe SSD, and so on in this order).

[0043] In this embodiment, the memory is the memory space of the CPU or the memory space of the storage acceleration system (in Figure 1 In the example, the memory space of the CPU is DDR1 on the CPU side, and the memory space of the storage acceleration system is DDR2 on the storage acceleration system side); wherein, the memory space of the CPU or the memory space of the storage acceleration system are uniformly addressed.

[0044] In this embodiment, the FPGA-based multi-channel NVMe SSD storage acceleration system further includes initialization before the read / write operation, and the initialization process is as follows:

[0045] First, a link is established between each NVMe Host controller and the corresponding NVMe SSD (a PCIE link is used between the NVMe Host controller and the NVMe SSD), so that each NVMe Host controller obtains the in-place status information and capacity information of the corresponding NVMe SSD;

[0046] Then, a link is established between the CPU and the NVMe Target controller, so that the CPU can obtain the in-place status information and capacity information of each NVMe SSD.

[0047] It should be noted that the specific initial process is:

[0048] NVMe Host initialization: This step includes PCIe link initialization and NVMe Host controller identification and initialization. After initialization is complete, all electronic disks are ready to receive I / O read and write commands.

[0049] NVMe Target initialization: This process is basically the same as NVMe Host initialization, except that the NVMe Target controller mentioned here belongs to the device side, while the host side is implemented by the CPU;

[0050] Loading the file system driver: After loading the driver, the host can identify the logical storage disk virtualized by the FPGA. The capacity is the sum of the capacities of all NVMe SSDs.

[0051] Format the file system: This step is to rebuild the data structure of the file system to manage data;

[0052] Mounting: This step connects the file system to the directory structure of the operating system so that the files and directories in the file system can be accessed.

[0053] It should be noted that when performing write operations on multi-channel NVMe SSDs:

[0054] When the user writes a file to each NVMe SSD, the CPU creates a write IO command in the memory (i.e., the CPU's memory space) and notifies the NVMe Target controller through the doorbell that there is a command to be executed. After receiving the doorbell, the NVMe Target controller retrieves the write IO command from the CPU memory, parses the write IO command, and sends the parsed write IO command to the Raid controller. The Raid controller converts the parsed write IO command into a second DDR control command and a second SSD control command corresponding to the number of NVMe Host controllers, transmits the second DDR control command to the DDR controller, and sends each second SSD control command to the NVMe Host controller one by one.

[0055] After receiving the second DDR control command, the DDR controller initiates a read operation on the memory and sends the read first data to the Raid controller. The Raid controller divides the first data into data blocks corresponding to the number of NVMe SSDs and sends each data block to the NVMe Host controller one by one.

[0056] After receiving the second SSD control command, each NVMe Host controller writes the received data block to the corresponding NVMe SSD, completing the write operation on the multi-channel NVMe SSD.

[0057] When performing a read operation on a multi-path NVMe SSD:

[0058] When the user reads a file from each NVMe SSD, the CPU creates a read IO command in the memory and notifies the NVMe Target controller through the doorbell that there is a command to be executed. After receiving the doorbell, the NVMe Target controller retrieves the read IO command from the CPU memory, parses the read IO command, and sends the parsed read IO command to the Raid controller. The Raid controller converts the parsed read IO command into a first DDR control command and a first SSD control command corresponding to the number of NVMe Host controllers, transmits the first DDR control command to the DDR controller, and sends each first SSD control command to the NVMe Host controller one by one.

[0059] After receiving the first SSD control command, each NVMe Host controller initiates a read operation on the corresponding NVMe SSD and sends the read second data to the Raid controller. The Raid controller merges the second data to obtain third data and sends the third data to the DDR controller.

[0060] After receiving the first DDR control command, the DDR controller writes the received third data into the memory, completing the read operation on the multi-channel NVMe SSD.

[0061] This FPGA-based multi-channel NVMe SSD storage acceleration system presents the device side to the CPU host side, allowing the CPU host side to perform synchronous read and write operations on the multi-channel NVMe SSDs on the device side, realizing storage resource pooling, facilitating the CPU host side's management and maintenance of the device side, and thus improving work efficiency (for the CPU host side, only one NVMe SSD needs to be managed and maintained); in this system, the number of NVMe SSDs depends on the number of links between the NVMe Host controller and NVMe SSDs in the system, and no PCIe hard-core controller and PCIe switch chip are required, making the overall system simpler and less costly; at the same time, the RAID function in this solution does not rely on an external CPU and is independently implemented by the storage acceleration system.

[0062] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. An FPGA-based multi-channel NVMe SSD storage acceleration system, characterized by: The storage acceleration system is used to read and write multiple NVMe SSDs. The storage acceleration system includes an NVMe Target controller, a Raid controller, a DDR controller, and an NVMe Host controller corresponding to the number of NVMe SSDs. The NVMe Host controller is connected to the NVMe SSDs in a one-to-one correspondence. The NVMe Target controller is connected to the CPU, and the DDR controller is connected to a memory. When performing read / write operations on a multi-way NVMe SSD: The CPU is used to create a read / write IO command and notify the NVMe Target controller through the doorbell that there is a command to be executed; The NVMe Target controller is used to extract the read / write IO command from the CPU after receiving the doorbell, parse the read / write IO command, and send the parsed read / write IO command to the Raid controller; The Raid controller is configured to, after receiving the parsed read / write IO command, convert the parsed read / write IO command and send the converted read / write IO command to the DDR controller and each NVMe Host controller respectively; and after receiving the first data sent by the DDR controller, divide the first data into a preset number of data blocks, and poll and send each data block to each NVMe Host controller; and after receiving the data blocks sent by each NVMe Host controller, merge each data block in the order sent by the Raid controller to obtain second data, and send the second data to the DDR controller; The DDR controller is configured to receive the second data sent by the Raid controller and write the second data into the memory after receiving the converted read IO command, and to read the first data from the memory and send the first data to the Raid controller after receiving the converted write IO command; Each of the NVMe Host controllers is used to read the data block from the corresponding NVMe SSD and send it to the Raid controller after receiving the converted read IO command, and is used to receive the data block sent by the Raid controller and write it to the corresponding NVMe SSD after receiving the converted write IO command.

2. The FPGA-based multi-channel NVMe SSD storage acceleration system according to claim 1, wherein: The Raid controller is used to convert the parsed read / write IO command after receiving it, and send the converted read / write IO command to the DDR controller and each NVMe Host controller respectively, in the following manner: After receiving the parsed read IO command, the Raid controller converts the parsed read IO command into a first DDR control command and a first SSD control command corresponding to the number of NVMe Host controllers, and sends the first DDR control command to the DDR controller, and sends each first SSD control command to the NVMe Host controller one by one, wherein the first DDR control command and each first SSD control command serve as the converted read IO commands from the Raid controller to the DDR controller and each NVMe Host controller respectively; After receiving the parsed write IO command, the Raid controller converts the parsed write IO command into a second DDR control command and a second SSD control command corresponding to the number of NVMe Host controllers, and sends the second DDR control command to the DDR controller, and sends each second SSD control command to the NVMe Host controller one by one, wherein the second DDR control command and each second SSD control command are used as the converted write IO commands sent by the Raid controller to the DDR controller and each NVMe Host controller respectively.

3. The FPGA-based multi-channel NVMe SSD storage acceleration system according to claim 2, wherein: The DDR controller receives the second data sent by the Raid controller and writes it into the memory after receiving the first DDR control command, and is used to read the first data from the memory and send it to the Raid controller after receiving the second DDR control command.

4. The FPGA-based multi-channel NVMe SSD storage acceleration system according to claim 2, wherein: After receiving the first SSD control command, each NVMe Host controller reads a data block from the corresponding NVMe SSD and sends it to the Raid controller, and is used to receive the data block sent by the Raid controller and write it to the corresponding NVMe SSD after receiving the second SSD control command.

5. The FPGA-based multi-channel NVMe SSD storage acceleration system according to claim 1, wherein: The size of the data block is the stripe size. The number of data blocks written or read by each NVMe SSD is the same. When each NVMe Host controller reads a data block, all NVMe Host controllers read the data block in the order in which the Raid controller polls and sends it.

6. The FPGA-based multi-channel NVMe SSD storage acceleration system according to claim 1, wherein: The parsed read IO command and write IO command each include an operation code, read or write data amount, LBA address, and memory address.

7. The FPGA-based multi-channel NVMe SSD storage acceleration system according to claim 6, wherein: The first DDR control command includes a write operation code, a first data amount, and a memory address; The second DDR control command includes a read operation code, a second data amount, and a memory address; Each of the first SSD control commands includes a read operation code, a third data amount, an LBA address, and a memory address, wherein a sum of the third data amounts in all the first SSD control commands is equal to the first data amount; Each of the second SSD control commands includes a write operation code, a fourth data amount, an LBA address, and a memory address, wherein a sum of the fourth data amounts in all the second SSD control commands is equal to the second data amount.

8. The FPGA-based multi-channel NVMe SSD storage acceleration system according to claim 1, wherein: The memory is the memory space of the CPU or the memory space of the storage acceleration system.

9. The FPGA-based multi-channel NVMe SSD storage acceleration system according to claim 1, wherein: The FPGA-based multi-channel NVMe SSD storage acceleration system also includes initialization before the read / write operation, and the initialization process is as follows: First, a link is established between each NVMe Host controller and the corresponding NVMe SSD, so that each NVMe Host controller obtains the in-place status information and capacity information of the corresponding NVMe SSD; Then, a link is established between the CPU and the NVMe Target controller, so that the CPU can obtain the in-place status information and capacity information of each NVMe SSD.

Images