NVMe interface expansion method based on FPGA

Abstract

The invention discloses an NVMe interface expansion method based on an FPGA, and belongs to the technical field of data storage. According to the method, an NVMe protocol is achieved through an FPGA chip, an NVMe interface is expanded through PCIe Core (RP), expansion interconnection of the high-density NVMe interface is achieved, a PCIe-Switch expansion chip is replaced, and the use requirements of high performance and high reliability of a flash memory storage product are met.

Description

Technical Field

[0001] The present invention belongs to the technical field of data storage, and particularly relates to an NVMe interface expansion method based on FPGA. Background Art

[0002] NVMe is a register-level interface used for communication between host software and non-volatile memory subsystems. It is mainly designed for solid-state storage devices (SSDs) using the PCIe interface in enterprise, data center, and client systems, and its goal is to maximize the potential of flash memory.

[0003] The existing technical solution is to expand the PCIe channel through an imported PCIe expansion chip. The technical principle of the existing solution is as Figure 1 shown. The CPU is connected to the PCIe expansion chip. The CPU is configured as the RC mode at the host end for expanding the peripheral interface; the PCIeSwitch expands the PCIe interface for connecting multiple PCIe devices and exchanging data with the PCIe EP device; the NVMeDisk operates in the PCIe slave mode, that is, it is configured as the EP, and the CPU performs IO communication.

[0004] The existing technical solution realizes data interconnection by connecting the host end and the control end through the PCIe physical link, parses the protocol at the transport layer, and transmits the parsed data to the application layer to complete data IO.

[0005] Physical layer: Data is transmitted between the NVMe hard disk and the CPU through the PCIe interface. The NVMe hard disk acts as a slave device (EP), and the CPU acts as a master device (RC); the PCIe interface is expanded between the hard disk and the CPU through a PCIe expansion chip to achieve more hard disk expansions. The physical layer encapsulates the physical layer data packets, and after encoding, decoding, and serial-parallel conversion, the data transmission of the physical layer is completed.

[0006] Data link layer: Receives and sends data packets from and to the physical layer, is responsible for receiving and sending link layer data packets for PCIe data transmission, and after error checking and CRC verification, etc., sends the link layer data to the physical layer or uploads it to the processing layer.

[0007] Processing layer: Parses or encapsulates the processing layer data, and parses or encapsulates the header and payload of the data packet according to the data format of the TLP.

[0008] Software layer: Performs requests and responses for business IO and read / write configurations.

[0009] Currently, in the application of large-capacity and high-density interfaces, the existing technical solutions still use imported PCIe-Switch chips to expand NVMe interfaces. In actual applications, the prices of these imported components are expensive; the IO chipset cannot meet the demand for high-density expansion of NVMe interfaces, so it cannot well satisfy high-performance flash storage products based on NVMe hard drives. Summary of the Invention

[0010] In view of this, the purpose of the present invention is to provide an NVMe interface expansion method based on FPGA.

[0011] An NVMe interface expansion method based on FPGA divides an FPGA chip into a slave device-side PCIe core facing the host computer, a command transmission scheduling module, an NVMe processing core, an XDMA module, and a master device-side PCIe core facing the NVMe hard drive;

[0012] When writing data, the CPU constructs a data payload and transmits it to the slave device-side PCIe core of the FPGA through the PCIe protocol data transmission format;

[0013] The slave device-side PCIe core submits a write request to the NVMe processing core;

[0014] The NVMe processing core receives a write command request from the CPU, submits the write command to the command transmission scheduling module; and the NVMe processing core parses the data payload and writes it into the memory module;

[0015] The command transmission scheduling module dispatches the write command to the master device-side PCIe core; the master device-side PCIe core reads the data payload from the memory module through the XDMA module according to the write command, then encapsulates the data payload in a data format and transmits and writes the data payload into the NVMe hard drive;

[0016] When reading data, the CPU constructs a read data command and transmits it to the slave device-side PCIe core of the FPGA through the PCIe protocol data transmission format;

[0017] The slave device-side PCIe core submits a read request command to the NVMe processing core;

[0018] The NVMe processing core receives a read command request from the CPU, submits the read command to the command transmission scheduling module, and the command transmission scheduling module dispatches the read command to the PCIe core at the main device end; the PCIe core at the main device end transmits the read command to the NVMe hard disk; after receiving the read instruction request, the NVMe hard disk prepares the data to be read out; the NVMe hard disk transfers the prepared data to the PCIe core at the main device end, and the PCIe core at the main device end unpacks the data capsule and sends it to the XDMA module; the XDMA module writes the data into the memory module, and then the NVMe processing core reads out the data in the memory module and uploads it to the CPU.

[0019] Preferably, the command transmission scheduling module includes a command submission work queue, a command end work queue, and a queue scheduling module; the PCIe core at the main device end includes a high-speed transmission interface, an RC processing module, and a protocol conversion module;

[0020] When writing data, the NVMe core processing module receives a write command request from the CPU, submits the write command to the command submission work queue in the command transmission scheduling module; the queue scheduling module schedules the commands in sequence according to the command sequence in the command submission work queue and dispatches them to the RC processing module; the RC processing module encapsulates the write instruction in the PCIe data packet format and transmits it to the NVMe hard disk through the PCIe link; after receiving the write instruction request, the NVMe hard disk notifies the RC processing module to write the data payload; the RC processing module reads the data payload from the memory module through the XDMA module, and then the protocol conversion module encapsulates the data payload in the data format and passes it to the RC processing module, and the RC processing module transmits and writes the encapsulated data payload into the NVMe hard disk through the high-speed transmission interface.

[0021] Preferably, when writing data, after a write instruction is completed, the NVMe hard disk replies with the end of the write instruction and uploads it to the NVMe processing core through the command end work queue, and the NVMe processing core uploads the end command to the CPU through the PCIe core at the slave device end to notify the end of the write command.

[0022] Preferably, when writing data, while uploading the write completion command to the CPU, the NVMe processing core constructs an end message, constructs a TLP packet through the RC core module and sends it to the NVMe hard disk to notify the NVMe hard disk of the end of the write command. Thus, the entire write process is completed.

[0023] Preferably, when reading data, the NVMe core processing module receives a read command request from the CPU, submits the read command to the command submission work queue in the command transmission scheduling module; the queue scheduling module schedules the commands in sequence according to the command sequence in the command submission work queue and dispatches them to the RC processing module; the RC processing module encapsulates the read instruction in the PCIe data packet format and transmits it to the NVMe hard disk through the PCIe link; after receiving the read instruction request, the NVMe hard disk prepares the data to be read out; the NVMe hard disk transfers the prepared data to the RC processing module, and the RC processing module unpacks the data capsule and sends it to the XDMA module; the XDMA module writes the data into the memory module according to the scheduling order, and then the NVMe processing core reads the data in the DDR and uploads it to the CPU.

[0024] Preferably, when reading data, after the NVMe finishes transmitting the read data, the NVMe hard disk replies to the end of the read instruction and uploads it to the NVMe processing core through the command end work queue, and the NVMe core uploads the end command to the CPU through the PCIe core EP at the slave device end to notify the end of the read command.

[0025] Preferably, when reading data, while uploading the read completion command to the CPU, the NVMe processing core constructs a read end message, constructs a TLP packet through the RC processing module and sends it to the NVMe hard disk to notify the hard disk of the end of the read command. Thus, the entire read process is completed.

[0026] Furthermore, it also includes a link negotiation method with the NVMe hard disk through the PICe protocol, specifically as follows:

[0027] Step 1, Detection stage: In the device detection stage, detect whether the PCIe device exists;

[0028] Step 2, Polling stage: Confirm the existence of the device and obtain basic information;

[0029] Step 3, Configuration stage: Set the configuration information of the device;

[0030] Step 4, Recovery stage: Perform link recovery and equalization;

[0031] Step 5, Working stage: After the link negotiation is completed, the device runs stably in this stage and starts the transmission of TLP packets.

[0032] The present invention has the following beneficial effects:

[0033] The present invention realizes the NVMe protocol by adopting an FPGA chip, expands the NVMe interface through the PCIe Core (RP), realizes the extended interconnection of high-density NVMe interfaces, replaces the PCIe-Switch expansion chip, and meets the high-performance and high-reliability usage requirements of flash memory storage products. Brief Description of the Drawings

[0034] Figure 1 is the principle of the prior art solution;

[0035] Figure 2 is Figure 1 the data transmission flow chart;

[0036] Figure 3 is the schematic diagram of the principle of the solution of the present invention;

[0037] Figure 4 is the schematic diagram of the physical layer connection;

[0038] Figure 5 is the link negotiation flow chart;

[0039] Figure 6 is the schematic diagram of the data link layer;

[0040] Figure 7 is the schematic diagram of the transport layer;

[0041] Figure 8 is the PCIe channel expansion model;

[0042] Figure 9 is the schematic diagram of the RC core processing flow. Detailed Description of the Invention

[0043] The present invention will be described in detail below with reference to the accompanying drawings and by way of examples.

[0044] The NVMe interface expansion technology based on the FPGA chip of the present invention uses the FPGA chip to expand the PCIe interface through independent technology, replacing the PCIe-Switch chip, so as to realize the NVMe interface expansion and meet the application of NVMe flash storage products.

[0045] The NVMe interface expansion technology solution based on the FPGA chip follows the OSI (Open System Interconnection Reference Model) model. As Figure 3 shown, in the FPGA chip, a slave device-side PCIe core (EP) facing the host computer, an NVMe processing core, a command submission work queue, a command end work queue, a queue scheduling module, an XDMA module, and a master device-side PCIe core (RP) facing the NVMe hard disk are divided.

[0046] The present invention realizes the interface expansion of PCIe (RC, master mode) through the FPGA chip. The expansion method is as follows: multiple PCIe RCs are implemented inside the FPGA; data communication with multiple PCIe EP devices is realized; the encapsulation and parsing of the NVMe protocol for connecting to the NVMe hard disk are realized; the interface expansion method of the PCIe RC is as Figure 8 shown.

[0047] The main device - side PCIe core (RP) for NVMe hard drives mainly includes the following three core modules:

[0048] 1. High - speed transmission interface;

[0049] Utilize the existing transmission channels (such as 10Gbps, 16Gbps, etc.) inside the FPGA as the physical channels for PCIe data transmission; Note: For example, PCIe3.0*8 means 8 PCIe transmission channels, and the transmission rate of each channel interface is 8GT / s; Connect to the processing module of PCIe RC to transmit TLP data packets;

[0050] 2. RC processing module:

[0051] 1) Connect downward to the high - speed transmission interface to transmit TLP data packets (including PCIe link discovery, rate negotiation, transmission control, and data transmission with the NVMe hard drive, etc.);

[0052] 2) Transmit TLP data packets upward to the protocol conversion module;

[0053] 3. Protocol conversion module:

[0054] 1) The protocol conversion module realizes the protocol conversion between PCIe and the NVMe hard drive;

[0055] 2) Downward, it is used to encapsulate NVMe protocol data in the form of "capsules" inside TLP data packets and transmit them to the NVMe hard drive through the high - speed transmission interface;

[0056] 3) Upward, it is used to parse the data packets of TLP into the data format of the NVMe protocol for upper - layer applications to parse and call.

[0057] The FPGA chip completes link negotiation with the NVMe hard drive through the PICe protocol (including detection of PCIe devices, rate negotiation, basic information configuration, etc.). As Figure 5 described, the negotiation process is as follows:

[0058] Step 1. Detection stage: In the device detection stage, detect whether the PCIe device exists;

[0059] Step 2. Polling stage: Confirm the existence of the device and obtain basic information;

[0060] Step 3. Configuration stage: Set the configuration information of the device;

[0061] Step 4. Recovery stage: Perform link recovery and equalization;

[0062] Step 5. Working stage: After link negotiation is completed, the device operates stably in this stage and starts to transmit TLP packets;

[0063] Taking writing data as an example, the working process of the extension method of the present invention is described.

[0064] Step 1: The CPU constructs a data payload and transmits it to the PCIe core (EP) of the slave device of the FPGA through the PCIe protocol data transmission format (TLP data packet).

[0065] Step 2: The PCIe core at the slave device end submits a write request to the NVMe processing core.

[0066] Step 3: The NVMe processing core receives the write command request from the CPU, submits the write command to the command submission work queue; and the NVMe processing core parses the data payload and writes it into the memory module (DDR).

[0067] Step 4: The queue scheduling module schedules the commands in sequence according to the command sequence in the command submission work queue and distributes them to the RC processing module.

[0068] Step 5: The RC processing module drives the protocol conversion module to encapsulate the write instruction (capsule) into the PCIe data packet format and transmits it to the NVMe hard disk through the high-speed transmission interface.

[0069] Step 6: After receiving the write instruction request, the NVMe hard disk notifies the RC processing module to write the data payload.

[0070] Step 7: The RC processing module reads the data payload from the memory module (DDR) through the XDMA module, and then the protocol conversion module encapsulates the data payload in the data format and passes it to the RC processing module. The RC processing module transmits the encapsulated data payload into and writes it into the NVMe hard disk through the high-speed transmission interface.

[0071] Step 8: After a write instruction is completed, the NVMe hard disk replies with the end of the write instruction and uploads it to the NVMe processing core through the command end work queue. The NVMe processing core uploads the end command to the CPU through the PCIe core (EP) of the slave device to notify the end of the write command.

[0072] Step 9: While uploading the write completion command to the CPU, the NVMe processing core constructs an end message, drives the protocol conversion module through the RC core module to construct a TLP packet and sends it to the NVMe hard disk through the high-speed transmission interface to notify the end of the write command to the NVMe hard disk. Thus, the entire write process is completed.

[0073] The working process of reading data:

[0074] Step 1: The CPU constructs a read data command and transmits it to the PCIe endpoint (EP) of the slave device of the FPGA in the PCIe protocol data transfer format (TLP packet).

[0075] Step 2: The PCIe endpoint of the slave device submits the read request command to the NVMe processing core.

[0076] Step 3: The NVMe processing core receives the read command request from the CPU and submits the read command to the command submission work queue.

[0077] Step 4: The queue scheduling module schedules the commands in sequence according to the command sequence in the command submission work queue and dispatches them to the RC processing module.

[0078] Step 5: The RC processing module drives the protocol conversion module to encapsulate the read instruction (capsule) in the PCIe packet format and transmits it to the NVMe hard disk through the high-speed transmission interface.

[0079] Step 6: After receiving the read instruction request, the NVMe hard disk prepares the data to be read.

[0080] Step 7: The NVMe hard disk transfers the prepared data to the RC processing module, and the RC processing module unpacks the data capsule and sends it to the XDMA module.

[0081] Step 8: The XDMA module writes the data into the memory module (DDR) in the scheduled order, and then the NVMe processing core reads the data in the DDR and uploads the data to the CPU according to the mapping relationship with the CPU memory address.

[0082] Step 9: After the NVMe finishes transmitting the read data, the NVMe hard disk replies with the end of the read instruction and uploads it to the NVMe processing core through the command end work queue. The NVMe processing core uploads the end command to the CPU through the PCIe endpoint (EP) of the slave device to notify the end of the read command.

[0083] Step 10: While uploading the read completion command to the CPU, the NVMe processing core constructs a read end message, drives the protocol conversion module through the RC processing module to construct a TLP packet and sends it to the NVMe hard disk through the high-speed transmission interface to notify the hard disk of the end of the read command. Thus, the entire read process is completed.

[0084] In summary, the above is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An FPGA-based NVMe interface expansion method, characterized in that, In the FPGA chip, a slave device - side PCIe core, a command transmission scheduling module, an NVMe processing core, an XDMA module facing the host computer, and a master device - side PCIe core facing the NVMe hard disk are partitioned; When writing data, the CPU constructs a data payload and transmits it to the slave device - side PCIe core of the FPGA through the PCIe protocol data transmission format; The slave device - side PCIe core submits the write request to the NVMe processing core; The NVMe processing core receives the write command request from the CPU and submits the write command to the command transmission scheduling module; And the NVMe processing core parses the data payload and writes it into the memory module; The command transmission scheduling module dispatches the write command to the master device - side PCIe core; the master device - side PCIe core reads the data payload from the memory module through the XDMA module according to the write command, then encapsulates the data payload in the data format and transmits and writes the data payload into the NVMe hard disk; When reading data, the CPU constructs a read data command and transmits it to the slave device - side PCIe core of the FPGA through the PCIe protocol data transmission format; The slave device - side PCIe core submits the read request command to the NVMe processing core; The NVMe processing core receives the read command request from the CPU, submits the read command to the command transmission scheduling module, and the command transmission scheduling module dispatches the read command to the master device - side PCIe core; the master device - side PCIe core transmits the read command to the NVMe hard disk; After receiving the read instruction request, the NVMe hard disk prepares the data to be read out; The NVMe hard disk transmits the prepared data to the master device - side PCIe core. The master device - side PCIe core unpacks the data capsule and sends it to the XDMA module; the XDMA module writes the data into the memory module, and then the NVMe processing core reads the data in the memory module and sends it to the CPU.

2. The method for expanding NVMe interface based on FPGA according to claim 1, characterized in that The command transmission scheduling module includes a command submission work queue, a command end work queue, and a queue scheduling module; the master device - side PCIe core includes a high - speed transmission interface, an RC processing module, and a protocol conversion module; When writing data, the NVMe core processing module receives the write command request from the CPU and submits the write command to the command submission work queue in the command transmission scheduling module; the queue scheduling module schedules the commands in sequence according to the command sequence in the command submission work queue and dispatches them to the RC processing module; The RC processing module encapsulates the write instruction in the PCIe data packet format and transmits it to the NVMe hard disk through the PCIe link; after receiving the write instruction request, the NVMe hard disk notifies the RC processing module to write the data payload; the RC processing module reads the data payload from the memory module through the XDMA module, then the protocol conversion module encapsulates the data payload in the data format and passes it to the RC processing module, and the RC processing module transmits and writes the encapsulated data payload into the NVMe hard disk through the high - speed transmission interface.

3. The method for expanding NVMe interface based on FPGA according to claim 2, characterized in that When writing data, after a write instruction is completed, the NVMe hard disk replies that the write instruction is completed, and uploads it to the NVMe processing core through the command end work queue. The NVMe processing core uploads the end command to the CPU through the PCIe core on the slave device side to notify the end of the write command.

4. The method for expanding an NVMe interface based on FPGA according to claim 3, characterized in that, When writing data, the write completion command is uploaded to the CPU, and the NVMe processing core constructs the end information. The RC core module constructs a TLP packet and sends it to the NVMe hard disk to notify the NVMe hard disk of the end of the write command. At this point, the entire write process is completed.

5. The method for expanding NVMe interface based on FPGA according to claim 2, wherein When reading data, the NVMe core processing module receives the read command request from the CPU and submits the read command to the command submission work queue in the command transmission scheduling module; the queue scheduling module schedules the commands in sequence according to the command sequence in the command submission work queue and dispatches them to the RC processing module; The RC processing module encapsulates the read instruction in PCIe data packet format and transmits it to the NVMe hard disk through the PCIe link; after receiving the read instruction request, the NVMe hard disk prepares the data to be read; The NVMe hard drive transmits the prepared data to the RC processing module, which decapsulates the data capsule and sends it to the XDMA module; the XDMA module writes the data to the memory module in the scheduling order, and then the NVMe processing core reads the data in the DDR and sends it to the CPU.

6. The method for expanding an NVMe interface based on FPGA according to claim 5, characterized in that When reading data, after NVMe completes the transmission of read data, the NVMe hard disk replies that the read command is completed, and uploads it to the NVMe processing core through the command end work queue. The NVMe core uploads the end command to the CPU through the PCIe core EP on the slave device side to notify the end of the read command.

7. The method for expanding an NVMe interface based on FPGA according to claim 6, characterized in that When reading data, the read completion command is uploaded to the CPU, and the NVMe processing core constructs the read end information. The RC processing module constructs a TLP packet and sends it to the NVMe hard disk to notify the hard disk of the end of the read command. At this point, the entire read process is completed.

8. The method for expanding NVMe interface based on FPGA according to claim 1, characterized in that It also includes the link negotiation method between FPGA and NVMe hard disk through PICe protocol, specifically: Step 1, Detection phase: Device detection phase, detect whether the PCIe device exists; Step 2, polling phase: confirm the existence of the device and obtain basic information; Step 3, Configuration phase: Set the configuration information of the device; Step 4: Recovery phase: restore and balance the link; Step 5, working phase: link negotiation is completed, the device operates stably in this phase, and starts transmitting TLP packets.

Images