A NVMe controller and its initialization and data reading and writing method

Abstract

The present invention belongs to PCIe devices, and relates to an NVMe controller and its initialization and data reading and writing methods, which include a PCIe module with a DMA unit, an address conversion unit ATU module, an SRAM and a device CPU. The present invention replaces the NVMe hardware control register with SRAM, polls the SRAM space content to obtain register change information and gives it to different modules for processing, and performs initialization, reset and other operations after polling the controller and register to perceive the change; after polling the doorbell register to perceive the change, the host storage command is obtained through the DMA unit and the command is parsed, and handed over to other modules for processing. The present invention does not require special hardware design and verification, reduces development costs and time, reduces hardware failure points and the possibility of hardware errors, simplifies the complexity of the system, and the NVMe controller is easy to modify, and the maintenance and update cost is low; the present invention does not need to develop a host-side driver, and the development workload is small.

Description

Technical Field

[0001] The present invention relates to an NVMe controller and an initialization and data reading and writing method thereof, and belongs to the technical field of PCIe equipment. Background Art

[0002] With the continuous development of technology, PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard, mainly used to expand the data throughput of the computer system bus and improve the communication speed of devices) devices have become an important part of modern computer systems and are widely used in storage, network, graphics processing and other fields. Due to the wide variety of PCIe devices and their different functions, it is crucial to manage them effectively.

[0003] In order to reduce the design workload, the NVMe-MI protocol (NVMe Management Interface, referred to as NVMe-MI protocol, is an interface specification for managing and monitoring NVMe storage devices) can be implemented, and the NVMe (Non-Volatile Memory Express, a high-performance storage interface and protocol designed to utilize the high bandwidth provided by the PCIe bus) ecosystem can be used for management.

[0004] Reusing the NVMe ecosystem has the following advantages:

[0005] 1. Standardization and compatibility: As part of the NVMe specification, the NVMe-MI protocol follows a unified standard and interface definition. This means that different PCIe devices can follow the same management protocol, simplifying the management process and improving system compatibility.

[0006] 2. Ecosystem sharing: Since NVMe has established a huge ecosystem and a broad application base in the storage field, the NVMe-MI protocol can reuse these existing ecological resources. This means that developers can use existing NVMe tools, drivers, and applications to manage PCIe devices, reducing development costs and barriers.

[0007] 3. Scalability and flexibility: The design of the NVMe-MI protocol allows for expansion and customization to meet the management requirements of different PCIe devices. At the same time, it can also be integrated with other management protocols and interfaces to achieve more comprehensive system management functions.

[0008] Traditional NVMe controllers rely on hardware implementation. The hardware needs to implement NVMe control module, doorbell module, command transceiver module, command processing module, and PRP (Physical Region Page) processing module. The NVMe control module is used by users to configure the NVMe controller. The NVMe controller generates corresponding logic to initialize, create queues, reset, and other operations, and notifies the device CPU through interrupts to perform corresponding processing. The doorbell module is used for users to interact with the device CPU. When users write to the doorbell module, the device will perform data transceiver processing or interrupt the device CPU for command processing. The command transceiver module obtains commands from the command queue in the host memory to the local memory according to the doorbell and configuration information triggered by the user, or sends the device's completion command to the command completion queue in the host memory. The command processing module parses the NVMe command for corresponding processing or sends it to the device CPU for processing. The PRP processing module obtains the host memory specified by the PRP information into the PRP buffer according to the instructions processed by the command processing module.

[0009] This approach requires hardware implementation, which has the characteristics of high performance and low latency, but also brings limitations in flexibility. Hardware logic implementation requires a dedicated hardware design and verification process, which not only increases development costs but may also delay product launch time.

[0010] Based on this, the present invention is proposed. Summary of the invention

[0011] The present invention realizes the function of hardware NVMe controller by combining software with general hardware, and needs to solve the following problems:

[0012] 1) The NVMe controller implements a register space for information exchange between the host and the NVMe device, so that the host can control the NVMe controller and obtain the NVMe controller status. The present invention uses a general hardware SRAM (Static Random-Access Memory) to replace this register space to allow the host and the software NVMe controller to communicate.

[0013] 2) SRAM only has the ability to store and access data, but no processing logic or notification mechanism. The user cannot actively notify the device CPU when writing to the SRAM address space, and the device CPU cannot make corresponding processing. The present invention uses the software thread on the device CPU to continuously poll and compare the content of the SRAM space, and performs corresponding processing after discovering data changes.

[0014] 3) The NVMe controller has the function of obtaining host memory to local cache and writing local cache data to host memory. The device CPU cannot communicate directly with the host memory, and cannot copy host memory and SRAM data to each other. The present invention uses the DMA unit (DMA, direct memory access, allowing hardware devices to directly transfer data with memory without CPU intervention) of the PCIe module in combination with SRAM to communicate between the device CPU and the host memory. SRAM is designed with two channels, channel one is reserved for the DMA unit of the PCIe module, and channel two is reserved for device CPU communication. The device CPU controls the DMA unit of the PCIe module to send and receive PCIe TLP packets (Transaction Layer Packet, which is the basic unit for data transmission in PCIe), uses channel one to copy the host memory to the local SRAM, and the CPU reads and writes the SRAM content through channel two.

[0015] 4) The NVMe controller has a command processing module and a PRP module. The hardware parses different commands for different processing or triggers hardware interrupts for processing. The present invention implements this part of the logic based on software. The commands are acquired by the DMA unit into the command queue of the local SRAM. The software reads the commands in sequence and processes them one by one. If the command contains PRP data, the device CPU parses the PRP data, and then controls the DMA unit to fetch data from the host memory to the SRAM for processing according to the parsed address.

[0016] The purpose of the present invention is to provide an NVMe controller, which is implemented by software and general hardware modules. Based on this NVMe controller, users can transfer NVMe, NVMe-MI and custom instructions to device firmware to control various PCIe devices. The advantage is that the present invention does not rely on hardware implementation, simplifies hardware design, reuses NVMe protocol for control, simplifies protocol design and implementation, and reduces R&D costs and cycles. The specific contents of the present invention are as follows:

[0017] In a first aspect, an NVMe controller includes a PCIe module with a DMA unit, an address translation unit ATU module, an SRAM, and a device CPU;

[0018] The PCIe module with DMA unit is used for communication between the device and the host and for information exchange between the device and the host. The PCIe module with DMA unit receives the host's read and write messages to the SRAM and forwards them to the SRAM; the control information received from the host is processed by the module itself or forwarded to the device CPU;

[0019] The DMA unit is controlled by the device CPU and actively initiates data transfer between SRAM and host memory;

[0020] The address translation unit ATU module is used to flexibly adjust the SRAM area occupied by the NVMe controller;

[0021] SRAM is used for data cache; SRAM has two channels, channel 1 and channel 2. Channel 1 is connected to the PCIe module with DMA unit, and channel 2 is connected to the device CPU;

[0022] The device CPU includes an NVMe controller hardware simulation module, an NVMe protocol communication module and an application control module. The NVMe controller hardware simulation module polls and monitors the registers and doorbell registers of the NVMe controller, and then hands them over to other modules for NVMe initialization, control, and communication processing;

[0023] The NVMe protocol communication module is used to receive NVMe commands, NVMe-MI commands and other custom commands from the host. The NVMe protocol communication module controls the DMA unit to read and write the host memory, obtain the host command, and transfer it to the corresponding module for processing according to the command type. After the processing is completed, the data to be returned is written to the host memory;

[0024] The application control module is responsible for initializing the NVMe controller and performing command processing.

[0025] In a second aspect, an NVMe controller initialization method comprises the following steps:

[0026] First, initialize the PCIe configuration space register, configure PCIe to NVMeEP mode, then initialize the SRAM value corresponding to the NVMe controller register, and initialize the basic properties related to the NVMe controller;

[0027] Then, the register monitoring thread is initialized, and the initial value of the SRAM area to be monitored is recorded first. After entering the loop, it is determined whether the register value has changed; if the register value has not changed, the loop continues until a command to shut down the device is received and exits;

[0028] When the value of the register changes, if the NVMe controller register controls the related register changes, the NVMe controller logic is executed; if the NVMe controller register doorbell register changes, it is handed over to the NVMe protocol communication module for processing.

[0029] Furthermore, the PCIe configuration space register is initialized by the application control module.

[0030] Furthermore, the basic attributes related to the NVMe controller include device type, number of queues or queue depth.

[0031] In a third aspect, a method for reading and writing data of an NVMe controller includes the following steps:

[0032] Step 1: The host places one or more commands to be executed in the next free submission queue slot in the host memory;

[0033] Step 2: The host writes the submission queue tail doorbell register, whose value is the value of the new submission queue tail entry pointer; the NVMe controller hardware simulation module monitors the change of the submission queue tail doorbell register value, senses that there is a new command to be processed, and hands the submission queue tail doorbell register value to the NVMe protocol communication module for processing;

[0034] Step 3, the NVMe protocol communication module controls the DMA unit to transfer the command in the submission queue slot to the command buffer of the SRAM;

[0035] Step 4, the NVMe protocol communication module reads the command of the SRAM and parses the command. If the command contains PRP information, NVMe parses the PRP information and transfers the memory page of the host referred to by the PRP to the buffer of the SRAM through the DMA unit;

[0036] Step 5: The NVMe protocol communication module parses the command and passes it to the application control module for corresponding business processing;

[0037] Step 6: After the command is executed, the NVMe protocol communication module generates a completion queue entry and stores it in the SRAM. Then the NVMe protocol communication module controls the DMA unit to store the completion queue entry in the next free slot in the associated completion queue of the host.

[0038] Step 7, the NVMe protocol communication module controls the PCIe module with DMA unit to generate an MSI-X interrupt to the host to indicate that there is a new completion queue entry that needs to be consumed and processed, and determines whether to generate an interrupt for each new completion queue entry according to the interrupt merging setting;

[0039] Step 8: The host consumes and processes new entries in the completion queue until it encounters an entry whose phase flag of a previously consumed entry is opposite to the value of the current completion queue entry.

[0040] Step 9: The host writes the bell register of the completion queue head, the device monitors the change of the completion queue head register, updates the local completion queue head cache, and ends.

[0041] The present invention replaces the NVMe hardware control register with SRAM, and the device CPU polls the SRAM space content to obtain register change information and gives it to different modules for processing. After the device CPU polls the register corresponding to the controller to perceive the change, it performs initialization, reset and other operations. After the device CPU polls the doorbell register to perceive the change, the device CPU obtains the host storage command by controlling the DMA unit and performs command parsing, and hands it over to the communication module processing module for processing.

[0042] Beneficial effects of the present invention:

[0043] 1. The present invention implements the NVMe controller by using general hardware combined with software, without the need for special hardware design and verification, thus reducing development costs and time. By reusing mature general modules, the possibility of hardware failure points and hardware errors is reduced, simplifying the complexity of the system. At the same time, the NVMe controller implemented by software is easy to modify and has low maintenance and update costs.

[0044] 2. The present invention implements the real NVMe protocol, and no host-side driver needs to be developed, which greatly reduces the development workload. Based on the NVMe protocol and NVMe-MI, it can be used as a universal management module, suitable for managing various devices. Each device can flexibly expand various device management functions and simplify the design workload of management software. The present invention brings important technological innovation and application value to the field of device management. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a general schematic diagram of the NVMe controller;

[0046] Figure 2 This is the initialization process of the NVMe controller described in the present invention;

[0047] Figure 3 The flowchart of the NVMe controller data reading and writing method. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Example 1

[0049] An NVMe controller such as Figure 1 As shown, it includes a PCIe module with a DMA unit, an address translation unit ATU (Address Translation Unit, a hardware component used to implement an address translation function in a computer system) module, SRAM, and a device CPU.

[0050] Figure 1 The "PCle with DMA" here stands for "PCIe module with DMA unit" and the "ATU" stands for "address translation unit ATU module".

[0051] The PCIe module with DMA unit is used for communication between the device and the host and information exchange between the device and the host. It receives the host's read and write messages to the SRAM and forwards them to the SRAM; it receives the host's control information and processes it in its own module or forwards it to the device CPU. Among them, the DMA unit is controlled by the device CPU and can actively initiate data transfer between the SRAM and the host memory.

[0052] The address translation unit ATU module is used to flexibly adjust the SRAM area occupied by the NVMe controller. With the address translation unit ATU module, the fixed area size memory can be occupied without occupying any non-continuous address SRAM area as needed according to the number of controllers required, thereby reducing unnecessary SRAM space occupation.

[0053] SRAM is used for data cache and serves as a bridge for communication between the host and the device CPU. It has two channels, channel one and channel two. Channel one is connected to the PCIe module with a DMA unit, and channel two is connected to the device CPU.

[0054] The device CPU is the core of the entire design, controlling the processing flow of the entire NVMe controller and also responsible for various business logics. It is mainly divided into:

[0055] NVMe controller hardware simulation module, NVMe protocol communication module, and application control module.

[0056] Among them, the NVMe controller hardware simulation module polls and monitors the registers and doorbell registers of the NVMe controller, and then hands them over to the NVMe protocol communication module for NVMe communication processing. The NVMe protocol communication module is used to receive the host's NVMe commands and the NVMe-MI commands above them, and also receives other custom commands. It controls the DMA unit to read and write the host memory, obtains the host command, and transfers it to the application control module for processing according to the command type. After processing, the NVMe protocol communication module writes the data to be returned to the host memory. The application control module is responsible for initializing the NVMe controller and processing commands. Example 2

[0057] The initialization process of the NVMe controller of Example 1 is as follows Figure 2 As shown:

[0058] First, you need to initialize the PCIe configuration space register through the application control module, configure PCIe to NVMeEP (endpoint in NVMe protocol) mode, and then initialize the SRAM value corresponding to the NVMe controller register, and initialize the basic properties related to the NVMe controller (such as device type, number of queues, queue depth, etc.). Then initialize the register monitoring thread, first record the initial value of the SRAM area to be monitored, and after entering the loop, determine whether the register value has changed. If the register value has not changed, continue to loop until the command to shut down the device is received and exit. If the register value has changed, if the NVMe controller register controls the related register changes, the NVMe controller logic is executed; if the NVMe controller register doorbell register has changed, it will be handed over to the NVMe protocol communication module for processing. Example 3

[0059] The process of NVMe controller data reading and writing method is as follows Figure 3 As shown, the following steps are included:

[0060] Step 1: The host places one or more commands to be executed in the next free submission queue slot in the host memory.

[0061] Step 2: The host writes the submission queue tail doorbell register, whose value is the value of the new submission queue tail entry pointer. The NVMe controller hardware simulation module monitors the change of the submission queue tail doorbell register value, senses that there is a new command to be processed, and passes the submission queue tail doorbell register value to the NVMe protocol communication module for processing.

[0062] Step 3: The NVMe protocol communication module controls the DMA unit to transfer the commands in the submission queue slot to the command buffer of the SRAM.

[0063] Step 4: The NVMe protocol communication module reads the command of the SRAM and parses the command. If the command contains PRP information, NVMe parses the PRP information and transfers the memory page of the host referred to by the PRP to the buffer of the SRAM through the DMA unit.

[0064] Step 5: The NVMe protocol communication module parses the command and passes it to the application control module for processing of corresponding services (such as device firmware upgrade, device peripheral control, device information acquisition, etc.).

[0065] Step 6: After the command is executed, the NVMe protocol communication module generates a completion queue entry and stores it in the SRAM. Then the NVMe protocol communication module controls the DMA unit to store the completion queue entry in the next free slot in the associated completion queue of the host.

[0066] Step 7: The NVMe protocol communication module controls the PCIe module with the DMA unit to generate an MSI-X (Message Signaled Interrupts eXtended) interrupt to the host to indicate that there is a new completion queue entry that needs to be consumed and processed. According to the interrupt merging settings, it is determined whether to generate an interrupt for each new completion queue entry.

[0067] Step 8. The host consumes and processes new entries in the completion queue until it encounters an entry whose phase flag of a previously consumed entry is opposite to the value of the current completion queue entry.

[0068] Step 9: The host writes the bell register of the completion queue head, and the device monitors the change of the completion queue head register, updates the local completion queue head cache, and the communication process ends.

[0069] exist Figure 3 In the figure, EP 0 is PCIe Endponit 0; BAR 0 is the register space pointed to by the first Base Address Register.

[0070] Controller corresponds to the controller. Doorrbells corresponds to the doorbell register, indicating that MSI-X (Message Signaled Interrupts eXtended) is an interrupt mechanism in the PCI Express (PCIe) specification, which is designed to provide more efficient and flexible interrupt handling capabilities than traditional interrupt methods. MSI-X triggers interrupts by using memory write operations instead of relying on traditional IRQ lines. The MSI-X Table is a core component of the MSI-X mechanism. It is a data structure stored in the device memory and is used to manage and configure MSI-X interrupts. The MSI-X Pending Table is an optional component in the MSI-X (Message Signaled Interrupts eXtended) mechanism that handles the suspension and delayed processing of interrupts. When a device generates an interrupt but cannot process it immediately, the interrupt information can be recorded in the PendingTable and processed later.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An NVMe controller, characterized in that: Includes PCIe module with DMA unit, address translation unit ATU module, SRAM and device CPU; The PCIe module with DMA unit is used for communication between the device and the host and for information exchange between the device and the host. The PCIe module with DMA unit receives the host's read and write messages to the SRAM and forwards them to the SRAM; the control information received from the host is processed by the module itself or forwarded to the device CPU; The DMA unit is controlled by the device CPU and actively initiates data transfer between SRAM and host memory; The address translation unit ATU module is used to flexibly adjust the SRAM area occupied by the NVMe controller; SRAM is used for data cache; SRAM has two channels, channel 1 and channel 2. Channel 1 is connected to the PCIe module with DMA unit, and channel 2 is connected to the device CPU; The device CPU includes an NVMe controller hardware simulation module, an NVMe protocol communication module and an application control module. The NVMe controller hardware simulation module polls and monitors the registers and doorbell registers of the NVMe controller, and then hands them over to other modules for NVMe initialization, control, and communication processing; The NVMe protocol communication module is used to receive NVMe commands, NVMe-MI commands and other custom commands from the host. The NVMe protocol communication module controls the DMA unit to read and write the host memory, obtain the host command, and transfer it to the corresponding module for processing according to the command type. After the processing is completed, the data to be returned is written to the host memory; The application control module is responsible for initializing the NVMe controller and performing command processing.

2. A method for initializing an NVMe controller, characterized in that: The method is implemented by an NVMe controller as claimed in claim 1, and comprises the following steps: First, initialize the PCIe configuration space register, configure PCIe to NVMeEP mode, then initialize the SRAM value corresponding to the NVMe controller register, and initialize the basic properties related to the NVMe controller; Then, the register monitoring thread is initialized, and the initial value of the SRAM area to be monitored is recorded first. After entering the loop, it is determined whether the register value has changed; if the register value has not changed, the loop continues until a command to shut down the device is received and exits; When the value of the register changes, if the NVMe controller register controls the related register changes, the NVMe controller logic is executed; if the NVMe controller register doorbell register changes, it is handed over to the NVMe protocol communication module for processing.

3. The NVMe controller initialization method according to claim 2, wherein: Initialize the PCIe configuration space registers through the application control module.

4. The NVMe controller initialization method according to claim 2, wherein: The basic attributes related to the NVMe controller include device type, queue number or queue depth.

5. A method for reading and writing data of an NVMe controller, characterized in that: The method is implemented by an NVMe controller as claimed in claim 1, and comprises the following steps: Step 1: The host places one or more commands to be executed in the next free submission queue slot in the host memory; Step 2: The host writes the submission queue tail doorbell register, whose value is the value of the new submission queue tail entry pointer; the NVMe controller hardware simulation module monitors the change of the submission queue tail doorbell register value, senses that there is a new command to be processed, and hands the submission queue tail doorbell register value to the NVMe protocol communication module for processing; Step 3, the NVMe protocol communication module controls the DMA unit to transfer the command in the submission queue slot to the command buffer of the SRAM; Step 4, the NVMe protocol communication module reads the command of the SRAM and parses the command. If the command contains PRP information, NVMe parses the PRP information and transfers the memory page of the host referred to by the PRP to the buffer of the SRAM through the DMA unit; Step 5: The NVMe protocol communication module parses the command and passes it to the application control module for corresponding business processing; Step 6: After the command is executed, the NVMe protocol communication module generates a completion queue entry and stores it in the SRAM. Then the NVMe protocol communication module controls the DMA unit to store the completion queue entry in the next free slot in the associated completion queue of the host. Step 7, the NVMe protocol communication module controls the PCIe module with DMA unit to generate an MSI-X interrupt to the host to indicate that there is a new completion queue entry that needs to be consumed and processed, and determines whether to generate an interrupt for each new completion queue entry according to the interrupt merging setting; Step 8: The host consumes and processes new entries in the completion queue until it encounters an entry whose phase flag of a previously consumed entry is opposite to the value of the current completion queue entry. Step 9: The host writes the bell register of the completion queue head, the device monitors the change of the completion queue head register, updates the local completion queue head cache, and ends.

Images