PCIE interface-based NVME communication system and method

Abstract

The application discloses a kind of NVME communication systems and methods based on PCIE interface, NVMe drive is carried in CPU side using operating system, and NVMe controller program is developed based on the PCIe communication interface of FPGA on FPGA end.Processing hard disk information required in NVMe drive loading process is stored in FPGA end, and data packet in accordance with the form of NVMe protocol stipulation and host end are communicated.A virtual SSD is generated on the FPGA side, and the CPU end can read and write the hard disk, the data packet received by the FPGA end is not stored, but algorithm processing is performed, and the CPU end can simultaneously read the operation to obtain the processing result.Compared with the prior art, the application based on PCIE board can be applied to more operating systems, plug and play and does not require driving, which provides convenience for the use of PCIE board.

Description

Technical Field

[0001] This invention belongs to the field of FPGA and CPU data communication interface application technology, and in particular relates to a method and apparatus for implementing an NVME communication interface based on a PCIe interface. Background Technology

[0002] With the rapid development of information technology, the requirements for data communication speeds are also increasing. Currently, most high-speed expansion cards rely on the PCIe interface for high-speed data transmission. PCIe (Peripheral Component Interconnect Express) is a high-speed serial computer expansion bus standard. Traditional communication between PCIe expansion cards and the CPU host uses PCIe-based Direct Memory Access (DMA) for data transmission, which presents the following practical problems:

[0003] 1) BLOCK DMA method is simple to implement, but it has a large operating system overhead and it is difficult to allocate a large contiguous block of memory.

[0004] 2) The Scatter Gather DMA method solves the Block DMA problem and greatly improves operational efficiency.

[0005] 3) Minimum Latency DMA mode: This mode does not involve interrupts or querying the board information, but only queries the data in the PC's local memory, so the system load and latency can be effectively controlled.

[0006] 4) Multi-DMA mode supports multiple independent data transfers to the PC via DMA. Each DMA is responsible for one or more transmit / receive channels. This method simplifies the structure of HDL, software, and drivers. There is no fixed data transfer pattern; the host PCIe driver and the FPGA-side PCIe device program must correspond one-to-one. Changing the operating system or operating system version requires PCIe driver adaptation, resulting in a significant development workload.

[0007] Non-Volatile Memory Express (NVME) is a protocol standard that runs on the PCIe interface. It implements the communication protocol between the host and the SSD, and is a PCIe-based storage device specification for high-speed solid-state drives (SSDs) as the host interface. The NVME protocol interface allows host software to communicate with the non-volatile memory subsystem, typically connecting to the PCIe interface as a register-level interface. Currently, mainstream systems such as Linux, Windows, and Unix all provide NVME drivers. There is a hierarchical relationship: NVME protocol <--- transaction layer <--- data link layer <--- physical layer. The NVME protocol, as the command and application layer protocol, is located at the top layer, with the next three layers being the PCIe interface portion. The PCIe specification defines the transport layer (or transaction layer), data link layer, and physical layer. Summary of the Invention

[0008] This solution aims to propose an NVME communication interface system and method based on the PCIe interface, realizing high-speed data communication between the CPU and FPGA based on the NVME protocol via the PCIe interface.

[0009] This invention is achieved using the following technical solution:

[0010] An NVMe communication system based on a PCIe interface includes a CPU host as an NVMe master device and an FPGA module as an NVMe slave device; the CPU host is connected to the FPGA module via a PCIe communication interface, and the FPGA module communicates with the CPU host according to data packets conforming to the NVMe protocol; wherein:

[0011] The CPU host further includes an NVME driver, which enables the CPU host to access and read / write operations on the FPGA module through the API interface provided by the NVME driver;

[0012] The FPGA module further includes a ROM memory, an NVMe controller, a virtual SSD hard disk, and an external DDR cache. It stores the hard disk information required during the NVMe driver loading process. The data written to the virtual SSD hard disk is sent to the FPGA module for algorithm processing to distinguish the data received by the NVMe controller from the data sent to it. The ROM memory is used to store the virtual SSD hard disk parameter information, which is the hard disk parameter information required during the NVMe driver loading process. The CPU host completes the identification and read / write operations of the virtual SSD in the FPGA module through the PCIe interface, thereby realizing data communication.

[0013] An NVME communication method based on a PCIe interface, the method specifically includes the following steps:

[0014] Step 1: By enumerating the PCIe communication interface, the PCIe communication interface space of the FPGA module is mapped to the address space of the computer. The CPU host completes the initialization of the NVMe controller by accessing the BAR0 space. Specifically, the CPU host initializes the NVMe controller according to the NVMe protocol requirements, performs register initialization configuration according to the NVMe protocol, and obtains the working status of the virtual SSD.

[0015] Step 2: The identification information of the virtual SSD being processed is stored in the ROM memory inside the FPGA module. The configuration space of the FPGA module's PCIe communication module is mapped to the address space on the computer side. The enumeration and configuration of the virtual SSD based on the FPGA module are completed. The virtual SSD being processed is initialized. Specifically, the FPGA module responds to different command completion queue instructions according to the different Admin command submission queues in the virtual SSD being processed, thus completing the virtual SSD initialization. After initialization, the virtual SSD initialization completion information is reported to the user, and the NVMe controller is ready to receive requests.

[0016] Step 3: The FPGA module receives the data extracted from the IO command submission queue, performs calculations, and sends the calculation results to the computer through the command completion queue. The CPU host submits IO command requests through the IO command submission queue. The IO command submission queue stores NVME commands, converting I / O command requests into NVME commands. The FPGA module obtains IO command requests through direct memory access, then receives and sends data. The processing results are uploaded to the command completion queue via direct memory access.

[0017] Compared with the prior art, the beneficial technical effects achieved by the present invention are as follows:

[0018] PCIe-based expansion cards are compatible with more operating systems, eliminate driver compatibility issues, and are plug-and-play, requiring no drivers. This provides convenience for the use of these expansion cards. Attached Figure Description

[0019] Figure 1 This is a topology diagram of an NVME communication system between a CPU and an FPGA.

[0020] Figure 2 This is a diagram showing the DB composition and operation assistant data structure of the NVME controller.

[0021] Figure 3 This diagram illustrates the data operation process between the CPU host, FPGA module, and NVME controller.

[0022] Figure 4 This is a flowchart of an NVME communication method between a CPU and an FPGA according to the present invention.

[0023] Figure 5 The process of initializing a virtual SSD. Detailed Implementation

[0024] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0025] like Figure 1 The diagram shows a topology of an NVME communication system between a CPU and an FPGA according to the present invention. The system includes a CPU host 10, an NVME driver 30, and an FPGA module 20. The CPU host 10 is connected to the FPGA module 20 via a PCIe bus. The CPU host 10 has an NVME driver 30, and the CPU host 10 and the FPGA module 20 are connected via the PCIe bus through the NVME driver 30. The NVME driver 30 executes the NVME communication protocol and uses the NVME driver included in the operating system.

[0026] The FPGA module 20 internally includes a ROM memory 301, an NVMe controller 302, and a virtual SSD hard disk 303, with an external DDR cache 304. The CPU host accesses the virtual SSD hard disk within the FPGA module through the API interface provided by the NVMe driver, performing read and write operations. The data communication between the CPU host and the virtual SSD hard disk is actually a data exchange between the CPU host and the NVMe controller within the FPGA module. Data written to the virtual SSD hard disk is not stored but sent to the FPGA module for algorithm processing; data read from the virtual SSD hard disk is the processed data result.

[0027] The ROM memory stores the virtual SSD hard drive parameter information. In this design, a real SSD component exists, requiring fixed hard drive information to be stored internally within the FPGA. When the host computer queries the hard drive information, it retrieves the information from the ROM and sends it back to the CPU.

[0028] The configuration parameters of a solid-state drive (SSD) include the total number of channels (NAND buses used for transmitting commands and data), the total number of CEs (chip selects, also called targets) within the NAND, the total number of LUNs, and the number of blocks contained in each LUN. The total number of blocks in the SSD is determined based on these parameters. The space loss rate of the SSD is calculated based on the total number of blocks and the number of ParityLUNs. It also includes the manufacturer, capacity, features, and log pages.

[0029] The CPU host and FPGA module each have a first PCIe communication interface 40 and a second PCIe communication interface 50. The first PCIe communication interface 40 is connected to the CPU host 10, and the second PCIe communication interface 50 is connected to the NVMe controller in the FPGA module for NVMe controller program development. The FPGA module stores the hard disk information required during NVMe driver loading and communicates with the CPU host using data packets conforming to the NVMe protocol. A virtual SSD hard disk is virtualized in the FPGA module, and the CPU host performs read and write operations on this hard disk. Data packets received by the FPGA module are not stored but processed... deal with The processing result is obtained by CPU host read operation.

[0030] In the NVME communication system between the CPU host and the FPGA of the present invention, the CPU host is the NVME master device and the FPGA is the NVME slave device.

[0031] like Figure 2 The diagram shows the composition of the NVME controller's database and the data structure of its operation assistant. It illustrates the database's composition and operational functions. The SQ Head DB and SQ Tail DB are updated by the controller, while the SQ Tail DB and CQ Head DB are updated by the CPU host. The database plays two roles in the data processing flow: remembering the header and footer of the SQ and CQ commands, and notifying the NVME controller of new commands requiring processing and command completion status.

[0032] like Figure 3 The diagram illustrates the data operation process between the CPU host, FPGA module, and NVME controller. This data operation process includes algorithm operation, SQ parsing operation, head doorbell maintenance operation, CQ framing operation, and tail doorbell maintenance operation. A detailed description follows:

[0033] The algorithm operation processes the data received by the FPGA module, distinguishing between data received and sent by the NVME controller. In this design, the algorithm logic involves bitwise inversion of the received data. The algorithm result is sent to DDR memory to store the Admin command submission queue (Adamin SQ) information during communication. This includes mapping the data transfer addresses between the CPU host and the NVME controller, such as the Admin SQ base address, Admin CQ base address, IO SQ base address, IO CQ base address, host memory address for reading / writing data, feature setting base address, and identifier base address. SQ parsing is performed to parse the received command submission queue SQ commands. Headdoorbell maintenance is performed to maintain the position of the headdoorbells on the CPU host and the NVME controller. SQ commands, CQ commands, and Doorbell information are transmitted via the high-speed data transfer link provided by the PCIe communication interface between the CPU host and the FPGA module. CQ framing is performed after receiving an SQ command from the CPU host, framing the CQ according to the data packet format specified by the NVME protocol, and sending it to the CPU host. Perform a tail doorbell maintenance operation to maintain the position of the CPU main unit and the tail doorbell. The operation result is saved to the ROM memory.

[0034] like Figure 4 The diagram shown is a flowchart of an NVMe communication method between a CPU and an FPGA according to the present invention. The CPU host mainly uses the PCIe interface to identify the virtual SSD on the FPGA side and communicate data.

[0035] Step 1: By enumerating the PCIe communication interface, the PCIe communication interface space of the FPGA module is mapped to the address space of the computer. The CPU host completes the initialization of the NVMe controller by accessing the BAR0 space. Specifically, the CPU host initializes the NVMe controller according to the NVMe protocol requirements, performs register initialization configuration according to the NVMe protocol, and obtains the working status of the virtual SSD.

[0036] Step 2: The identification information of the virtual SSD being processed is stored in the ROM memory inside the FPGA module. The configuration space of the FPGA module's PCIe communication module is mapped to the address space on the computer side. The enumeration and configuration of the virtual SSD based on the FPGA module are completed. The virtual SSD being processed is initialized. Specifically, the FPGA module responds to the commands in the different Admin command submission queues (Adamin SQ) in the virtual SSD being processed with the commands in the different command completion queues (CQ) to complete the virtual SSD initialization. After the initialization is completed, the virtual SSD initialization completion information is reported to the user. The NVMe controller is ready to receive requests.

[0037] The FPGA's internal ROM memory stores virtual SSD parameters, including the total number of channels, total number of CEs, total number of LUNs, and total number of Bolts. These parameters create a virtual SSD on the FPGA side for data communication with the host. The NVME controller's stored data is mainly divided into two parts: one part defines the overall attributes of the NVME controller, and the other part stores the head and tail DB registers for each queue.

[0038] This step also includes: before establishing the above queues, the CPU host obtains the number N of CPU cores running in the current system, consults the limit M of the number of queues in the NVME controller through the Set Feature command, selects the smaller value of M and N (MIN(N,M)) to create the IOSQ queue and CQ queue on the CPU host, and notifies the NVME controller;

[0039] Step 4: The FPGA module receives data retrieved from the I / O command submission queue (IOSQ), performs calculations, and sends the results to the computer via the command completion queue (CQ). The I / O command submission queue stores NVME commands, which are the basic unit of communication between the host and the NVME controller; I / O requests must also be converted into NVME commands. The CPU host submits I / O requests through the I / O command submission queue (IOSQ). The FPGA module acquires I / O requests via DMA, and then uses DMA to receive and send data; the processing results are uploaded via the command completion queue (CQ), which uses DMA for communication.

[0040] like Figure 5 The image shows the virtual SSD initialization process. The specific process description is as follows:

[0041] Step 2-1, PCIe Device Enumeration: During system startup, the BIOS enumerates the entire PCI bus and then transmits the scanned devices to the operating system via ACPI tables. When the operating system loads, the PCI Bus driver reads the Header Config space of each PCI device based on this information and obtains a characteristic value from the class code register. The class code is the sole basis for the PCI bus to select which driver to load the device. The class code defined in the NVMe Spec is 010802h, and the class code in the Controller PCIe Header inside the NVMe SSD is set to 010802h.

[0042] Step 2-2: Configure the PCIe registers according to the system configuration, including power management settings, interrupt settings, etc.

[0043] Steps 2-3: Read the NVME controller's control capabilities (NVMECAP): Determine the pages, doorbells, physical addresses, etc., supported by the VMe controller;

[0044] Steps 2-4: Read the NVME controller reset completion status (CSTS.RDY).

[0045] Steps 2-5: Determine if CSTS.RDY = 0 is satisfied;

[0046] If not, return to step 2-4; if yes, continue to step 2-6.

[0047] Steps 2-6: Configure the Admin queue, including Admin Queue Attributes (AQA), Admin Submission Queue Base Address (ASQ), and Admin Completion Queue Base Address (ACQ). The Admin Submission Queue is used by the host to submit Admin commands to the NVME device; the Admin Completion Queue is used by the NVME device to return command completion status to the host.

[0048] Steps 2-7: Configure NVME controller parameters, including setting the arbitration mechanism, initializing memory page size, and selecting IO request commands;

[0049] Steps 2-8: Configure CC.EN=1, which enables the controller;

[0050] Step 2-9: Determine if CSTS.RDY = 1 is satisfied. This expression means "waiting for the NVME controller to indicate that it is ready to process the command".

[0051] If not, return to step 2-8; if yes, proceed to step 2-9.

[0052] Steps 2-9: Obtain NVME controller cooperation, determine namespace configuration, and use the application to issue identification commands and specify controller data structures to determine the controller configuration. The master controller should determine the configuration of each namespace by issuing identification commands and specifying namespace data structures.

[0053] Step 2-10: The master controller should configure the MSI and MSI-X registers after setting the number of queue function identifiers and determining the number of IOSQ and IOCQ by setting the function instructions.

[0054] 1. Allocate the number of IOCQs according to the system configuration requirements and the number of controllers required, and use the create IOCQ command to allocate IOCQs;

[0055] 2. Allocate the number of IOSQs according to the system configuration requirements and the number of controllers required, and use the Create IOSQ command to allocate IOSQs.

[0056] After the above process, a drive letter is obtained on the CPU host side, and the CPU host communicates with the FPGA module through this drive letter. The FPGA module uses the hard disk information stored in its local ROM memory to cooperate with the host in registering and using the virtual SSD device. However, the FPGA module itself does not have virtual SSD storage space, i.e., a virtual SSD hard disk. The CPU host reads and writes data to the virtual SSD hard disk, which is actually communicating with the virtual SSD hard disk inside the FPGA module. The data sent by the CPU host is processed inside the FPGA, and the data read by the CPU host is the result data processed by cryptographic algorithms.

[0057] Before establishing the IO command submission queue and the IO command completion queue, the host obtains the number of CPU cores N currently running in the system, and then consults the NVME controller via the SetFeature command to determine the limit M of the number of queues. The host selects the MIN(N,M) value to create the queue and simultaneously notifies the NVME controller.

[0058] The specific process for establishing the IO command submission queue and the IO command completion queue is as follows:

[0059] a. Initialize the AdminQueue properties (AQA), AdminQueue base address (ASQ), and completion queue base address register;

[0060] b. Configure the size of the IO command submission queue (IOSQ) "CC.IOSQES" and the size of the IO command completion queue (IOCQ) "CC.IOCQES";

[0061] c. Configure the controller (CC), including defining the element size of the IO SQ queue and IOCQ queue, power-off status alerts, arbitration mechanism, memory page size, supported IO instruction set, and enabling;

[0062] d. Set up the IO command submission queue and IO command completion queue by issuing queue attribute commands. The CQ setting of the feature indicates the number of IO command submission queues and IO command completion queues.

[0063] e. Set the maximum number of queues each can support and whether the queues need to be physically contiguous;

[0064] f. Create an I / O command completion queue (IOCQ);

[0065] g. Create an IO command submission queue (IOSQ).

[0066] The response commands of the NVME controller are described as follows:

[0067] The NVMe controller's response commands are divided into Admin commands and IO commands. Admin commands are transmitted through the Admin Command Submission Queue (Admin SQ) and Admin Command Completion Queue (Admin CQ), primarily responsible for managing the NVMe controller, and also include some control instructions. IO commands are mainly transmitted through the IO Command Submission Queue (IO SQ) and IO Command Completion Queue (IO CQ), primarily responsible for completing data transmission.

[0068] For the received Admin command, it is necessary to determine whether it is a GET command or a SET command. If it is a SET command, the command is stored in the DDR cache, and the same configuration command overwrites the cache. If it is a GET command, the required information is obtained from the FPGA local ROM and DDR cache and sent to the CPU host according to the NVME protocol.

[0069] During communication between the CPU host and the NVMe controller, the FPGA module replies with the configuration management information required by the Admin command based on the information in its local ROM memory, and virtualizes a virtual SSD hard disk on the FPGA side. This ensures that the CPU host can perform read and write operations on the virtual SSD on the FPGA side. Table 1 shows the Admin command and its processing method.

[0070] Table 1

[0071]

[0072]

[0073] Some specific implementation steps of this invention are described below:

[0074] The steps for submitting and completing the Admin command on the NVME controller are as follows:

[0075] I. The CPU host places multiple commands to be processed into the next Admin command submission queue (Admin SQ);

[0076] II. The CPU host updates the DB register at the tail of the commit queue with the new commit queue tail pointer to inform the controller that a new command has been committed and needs to be processed.

[0077] III. The NVME controller reads commands from the Admin command submission queue (Admin SQ) to the NVME controller. The arbitration mechanism is used to determine which submission queue the controller should execute.

[0078] IV. The controller handles the execution of the next command, which may be executed out of order;

[0079] V. After the command is executed, the controller places it in the associated Admin command completion queue (Admin CQ). In the CQ, the NVME controller records the position of the submission queue head pointer SQHD. Each new CQ will reverse the Phase Tag to indicate that this CQ is new.

[0080] VI. The NVME controller sends an interrupt to the host, informing the host that there is a new CQ to be processed;

[0081] VII. The CPU host processes CQ, including generating actions in response to errors;

[0082] VIII. The CPU host writes the completion queue head DB register (CQHD) to indicate that the CQ has been processed, and updates the associated completion queue head DB register (CQHD) after processing the CQ.

[0083] II. The steps for constructing the response command of the NVME controller are as follows:

[0084] Build a command submission queue (SQ) instruction on the CPU host side:

[0085] a) Configure CDW0.OPC with the appropriate command;

[0086] b) Is CDW0.FUSE a fused op?

[0087] c) CDW0.CID and SQ combine to form a unique cmd id;

[0088] d)NSID

[0089] e)MPTR, record the Meta Region offset.

[0090] f) PRP1 / 2 records the data transmission src / dst

[0091] g)CDW10-15cmd spec info

[0092] The CPU host software writes to the corresponding commit queue DB register (SQxTDBL) to submit multiple cmd processes. A write operation to the commit queue DB register triggers the controller to retrieve SQs from the commit queue, and the NVME controller will return the most recently processed SQ in the completion item CQ.

[0093] III. The NVME controller generates an interrupt upon command completion and processes the completed command. The process steps are as follows:

[0094] a) DW2.SQID indicates the commit queue ID;

[0095] b) DW3.CID indicates completion of cmd;

[0096] c) DW3.SF indicates the completion status of cmd;

[0097] d) Errors will be logged in DW3.SF.

[0098] The host software indicates the available completion queues (CQ) by updating the corresponding completion queue Head DB register (CQyHDBL). Updating CQyHDBL also clears the corresponding interrupts.

[0099] IV. The response process of the NVME control side to IO requests is as follows:

[0100] After the NVME controller is initialized, the data transfer addresses between the CPU host and the NVME controller are mapped. These addresses include the base addresses of the admin SQ and ADMIN CQ, the base addresses of the IO SQ and IO CQ, the host memory address of the data to be read or written, the base address of setting features, and the base address of the identifier, etc.

[0101] After the NVME controller initialization is complete, the host creates IO SQs and IO CQs. The host and NVME controller communicate with each other via commands for more granular information and exchange data using a queue-based direct memory access (DMA) method. The queue stores NVME commands, which are the basic unit of communication between the host and the NVME controller; application I / O requests are also converted into NVME commands. The host submits IO requests via SQs; the FPGA acquires IO requests via DMA, and then receives and sends data via DMA; processing results are uploaded via CQs, which also use DMA for communication.

[0102] The IO command write process for the NVME controller includes the following steps:

[0103] 1. The FPGA module queries the value of the SQ DB register to determine if the register value has been updated. If it has, the FPGA module updates its local SQ Head DB register. If it has not been updated, the module continues to query the value of the SQ DB register.

[0104] 2. Update the local SQ Head DB register of the FPGA module to indicate the position of the current command operation and to compare it with the queried SQ DB to determine whether there is an SQ DB register update.

[0105] 3. If local storage space is insufficient, after receiving a data transmission request from the host, the FPGA module needs to determine if the data storage space is insufficient. If it is insufficient, it means there is storage space available, and the FPGA module can retrieve data from the host memory via DMA. If it is full, it needs to wait for local data processing before continuing the DMA operation.

[0106] 4. Data is read via DMA. The FPGA module reads data from the host memory based on the address information of SQ.

[0107] 5. Write completion information to CQ and notify the host SQ that the information processing is complete through CQ.

[0108] 6. Update the CQTailDB registers, updating the local CQ Tail DB registers.

[0109] 7. Notify the host that the command has been completed and update the CQ Head DB.

[0110] The data reading process for the NVME controller's IO command is as follows:

[0111] a) The host prepares a read command and places the command into the SQ in memory.

[0112] b) The host notifies the NVME that there are new commands to process. SQyTDBL (Submission Queue y TailDoorbell) is the control register information. The host and FPGA communicate through BAR0, and the host updates the value of the control register in BAR0.

[0113] c) The NVMe controller receives the register update message and reads a 64-byte SQ from the host via DMA. From the read SQ, it obtains the command address, queue and command ID, operation type (OPC), data address, and the SSD's Device ID in the system.

[0114] d) The NVME controller processes the SQ command in step 3 and writes the data to be sent to the specified memory address using DMA.

[0115] e) The NVME controller summarizes the command execution status into a record and adds it to the CQ. At this time, the NVME controller adds a 16-byte record to the host's CQ memory queue.

[0116] f) The NVME controller notifies the host to process the record in CQ via an interrupt.

[0117] g) CPU host processes the contents of CQ.

[0118] h) The CPU updates the Completion Queue and Head Doorbell information in memory. The corresponding DB information in the control registers will also be updated.

[0119] Explanation of abbreviations used in this invention:

[0120] PCIe (Peripheral Component Interconnect express): A high-speed serial computer expansion bus standard.

[0121] NVMe (Non-Volatile Memory express): A PCIe-based storage device specification for high-speed solid-state drives (SSDs) host interfaces.

[0122] FPGA (Field-Programmable Gate Array): Field-programmable gate array.

[0123] SQ (Submissions Queue): The command submission queue, used by the host to submit commands to the NVMe device.

[0124] CQ (Completion Queue): The command completion queue is used by NVMe devices to return command completion status to the host.

[0125] DB (Doorbell): Doorbell.

[0126] CAP—Controller Capability—defines the maximum and minimum memory page size, supported I / O instruction set, DB register step size, wait time limits, arbitration mechanism, whether the queue is physically contiguous, and queue size.

[0127] VS – Version Number, defines the version number of the controller's implementation of the NVMe protocol.

[0128] INTMS—Interrupt Mask. Each bit corresponds to an interrupt vector. This register is invalid when using MSI-X interrupts.

[0129] INTMC – Interrupt enabled. Each bit corresponds to an interrupt vector. This register is invalid when using MSI-X interrupts.

[0130] CC – Controller Configuration – defines the I / OSQ and CQ queue element size, shutdown status alerts, arbitration mechanism, memory page size, supported I / O instruction sets, and enable settings.

[0131] CSTS – Controller Status, including shutdown, controller fatal error, and ready status.

[0132] AQA—Admin queue attributes, including SQ size and CQ size.

[0133] ASQ—Admin SQ base address.

[0134] ACQ—Admin CQ base address.

[0135] SQ Head DB — Command submission queue header data.

[0136] SQ Tail DB — Command to submit data to the tail of the queue.

[0137] CQ Head DB — Command to complete queue head data.

[0138] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. An NVMe communication method based on a PCIe interface, comprising an NVMe communication system based on a PCIe interface to implement NVMe communication, characterized in that, The system includes a CPU host as an NVMe master device and an FPGA module as an NVMe slave device. The CPU host is connected to the FPGA module via a PCIe communication interface, and the FPGA module communicates with the CPU host according to data packets conforming to the NVMe protocol. The CPU host further includes an NVMe driver, which enables the CPU host to access and perform read / write operations on the FPGA module through its API interface. The FPGA module further includes a ROM memory, an NVMe controller, a virtual SSD, and an external DDR cache. Data written to the virtual SSD is not stored but sent to the FPGA module for algorithmic processing to distinguish between data received and sent by the NVMe controller. The ROM memory stores virtual SSD parameter information, which is required for the NVMe driver loading process. The CPU host identifies and performs read / write operations on the virtual SSD in the FPGA module via the PCIe interface, thereby achieving data communication. During the communication between the CPU host and the NVME controller, the FPGA module replies with the configuration management information required by the Admin command based on the information in the local ROM memory, and virtualizes a virtual SSD hard disk on the FPGA side to ensure that the CPU host can perform read and write operations on the virtual SSD on the FPGA side. The data operation process between the CPU host, FPGA module, and NVME controller includes algorithm operation, SQ parsing operation, head doorbell maintenance operation, CQ framing operation, and tail doorbell maintenance operation. The algorithm operation is used to process the data received by the FPGA module, distinguishing the data received by the NVME controller from the data sent by the NVME controller. The algorithm logic is to invert the received data bit by bit, and the algorithm operation result is sent to the DDR memory. Specifically, the following steps are included: Step 1: By enumerating the PCIe communication interface, the PCIe communication interface space of the FPGA module is mapped to the address space of the computer. The CPU host completes the initialization of the NVMe controller by accessing the BAR0 space. Specifically, the CPU host initializes the NVMe controller according to the NVMe protocol requirements, performs register initialization configuration according to the NVMe protocol, and obtains the working status of the virtual SSD. Step 2: The identification information of the virtual SSD being processed is stored in the ROM memory inside the FPGA module. The configuration space of the FPGA module's PCIe communication module is mapped to the address space on the computer side. The enumeration and configuration of the virtual SSD based on the FPGA module are completed. The virtual SSD being processed is initialized. Specifically, the FPGA module responds to different command completion queue instructions according to the different Admin command submission queues in the virtual SSD being processed, thus completing the virtual SSD initialization. After initialization, the virtual SSD initialization completion information is reported to the user, and the NVMe controller is ready to receive requests. Step 2 also includes the following processing: The CPU host obtains the number N of CPU cores currently running in the system, consults the limit M of the number of queues in the NVME controller through the Set Feature command, selects the smaller value between M and N, MIN(N,M), creates the IO SQ queue and CQ queue on the CPU host, and notifies the NVME controller. Step 3: The FPGA module receives the data retrieved from the IO command submission queue, performs calculations, and sends the calculation results to the computer through the command completion queue. The IO command submission queue stores NVME commands, which are the basic unit of communication between the host and the NVME controller; I / O requests must also be converted into NVME commands. The CPU host submits IO command requests through the IO command submission queue. The FPGA module obtains the IO command requests through direct memory access, and then receives and sends data through direct memory access. The processing results are uploaded through the command completion queue, which uses direct memory access for communication. Step 2, the virtual SSD initialization process specifically includes the following steps: Step 2-1, PCIe device enumeration: When the system starts, the BIOS enumerates the entire PCI bus and then passes the scanned devices to the operating system through ACPI tables; when the operating system loads, the PCI Bus driver reads the Header Config space of each PCI device based on this information and obtains a characteristic value from the class code register. The class code is the only basis for the PCI bus to select which driver to load the device. The class code defined by the NVMe Spec is 010802h, and the class code in the PCIe Header of the Controller inside the NVMe SSD will be set to 010802h. Step 2-2: Configure the PCIe registers according to the system configuration, including at least power management settings and interrupt settings; Steps 2-3: Read the control capabilities of the NVMe controller: Determine the pages, doorbells, and physical addresses supported by the NVMe controller; Steps 2-4: Read the NVME controller reset completion status; Steps 2-5: Determine if CSTS.RDY=0 is satisfied; If not, return to step 2-4; if yes, continue to step 2-6. Steps 2-6: Configure the Admin queue, including Admin queue attributes, Admin command submission queue base address and Admin command completion queue base address. The Admin command submission queue is used by the host to submit Admin commands to the NVME device, and the Admin command completion queue is used by the NVME device to return the command completion status to the host. Steps 2-7: Configure NVME controller parameters, including setting the arbitration mechanism, initializing memory page size, and selecting IO request commands; Steps 2-8: Configure CC.EN=1, which enables the controller; Step 2-9: Determine if CSTS.RDY=1 is satisfied. This expression means "waiting for the NVME controller to indicate that it is ready to process the command". If not, return to step 2-8; if yes, proceed to step 2-10. Steps 2-10: Obtain NVME controller cooperation, determine namespace configuration, and use the application to issue identification commands and specify controller data structures to determine the controller configuration. The master controller should determine the configuration of each namespace by issuing identification commands and specifying namespace data structures. Step 2-11: The master controller should configure the MSI and MSI-X registers after setting the number of queue function identifiers and determining the number of IO SQ and IO CQs by using function instructions. The number of IO CQs is allocated based on system configuration requirements and the number of controllers required, using the IO CQ creation command. The number of IO SQs is allocated based on system configuration requirements and the number of controllers required, and IO SQs are allocated using the Create IO SQ command.

2. The NVME communication method based on a PCIe interface as described in claim 1, characterized in that, The NVME controller's response commands are divided into Admin commands and IO commands. Admin commands are transmitted through the Admin command submission queue and the Admin command completion queue, and are responsible for managing the NVME controller, including control commands. IO commands are transmitted through the IO command submission queue and the IO command completion queue, and are responsible for completing data transmission.

3. The NVME communication method based on a PCIe interface as described in claim 1, characterized in that, The NVME controller's stored data is divided into two parts: one part defines the overall attributes of the NVME controller, and the other part consists of the head and tail DB registers of each queue.

Images