An NVMe acceleration system, method, device, and readable medium
By coordinating the design of the PCIe controller, memory controller, and NVMe accelerator, the IO processing flow of NVMe hard drives is optimized, solving the problem of CPU becoming a bottleneck, achieving efficient IO processing and hardware resource optimization, and reducing hardware costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-23
- Publication Date
- 2026-03-27
AI Technical Summary
When using NVMe SSDs, the increased IO processing power of existing technologies causes the CPU to become the system bottleneck. Traditional methods result in high latency and wasted hardware resources, making it difficult to effectively utilize the high IOPS performance of NVMe SSDs.
By co-designing the PCIe controller, memory controller, NVMe accelerator, and CPU, management command queues, IO command queues, and IO completion queues are created. Hardware acceleration is used to accelerate certain parts of the NVMe protocol, reducing CPU load and optimizing the IO processing flow.
Reduce I/O processing latency, decrease queue cache depth, avoid wasting hardware resources, reduce hardware costs, break through CPU I/O processing bottlenecks, and improve system performance.
Smart Images
Figure CN115562574B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of computers, and more particularly to an NVMe acceleration system, method, device and readable medium. BACKGROUND
[0002] NVMe (Non-Volatile Memory Host Controller Interface Specification) hard disks are increasingly widely used in storage systems such as storage servers due to excellent performance. Compared with traditional HDD hard disks using interfaces such as SATA / SAS, the advantages of NVMe hard disks mainly include the following aspects: due to the use of a higher-speed PCIe as an underlying transmission protocol by the NVMe protocol, the transmission bandwidth is increased from hundreds of MB / s to several GB / s; the read / write delay is reduced from the order of 10 ms to tens of us for solid-state storage media compared with magnetic media; due to the optimization design of the NVMe protocol for the IO interaction process of solid-state media, high concurrency is supported, and the IO processing capacity of the hard disk is increased from hundreds of IOPS to hundreds of thousands of IOPS.
[0003] Using NVMe hard disks, the storage system can break through the traditional performance bottleneck and achieve a substantial improvement, but at the same time, such a qualitative leap in performance also brings new challenges to system design.
[0004] One of the challenges is the increase in IOPS. Since the IOPS of traditional hard disks is very low, even in a storage array containing a large number of hard disks, the total IOPS can only reach tens of thousands to hundreds of thousands. The IOPS provided by a storage system using NVMe hard disks can be as high as several million to tens of millions. Therefore, in order to fully exert the IO capability of the hard disk in system performance, the IO processing capacity of the system needs to be at least 100 times higher than that of the traditional method. Although the method of hardware acceleration for different functions of the entire storage system has been widely introduced, even for a dedicated hardware chip, such high IO processing capacity still brings considerable challenges to design.
[0005] The disk manager generally needs to implement an NVMe protocol stack, which defines a large number of complex commands. Some of these commands have relatively complex interaction processes, such as asynchronous event commands; some have complex data structures, such as log pages, which are generally organized in the form of a linked list; and some have complex execution processes, such as abort commands and queue delete commands. Therefore, the traditional method is generally implemented by an embedded CPU and software. The software can implement complex process control and has high flexibility, but when the system IOPS is high, the array manager will make a large number of calls to the disk controller, and considering that the disk controller also needs to form the data structure required by the disk read / write IO from the SoC management data, as well as the interaction between the disk and the disk controller after the completion of the task, the embedded CPU will bear a huge burden and become a bottleneck of the system. Summary of the Invention
[0006] In view of this, the purpose of this invention is to propose an NVMe acceleration system, method, device, and readable medium. By using the technical solution of this invention, hardware can accelerate certain parts of the NVMe protocol, which avoids the high cost of implementing a complex NVMe protocol stack in hardware, breaks through the bottleneck of CPU IO processing, avoids the high latency and access overhead of traditional methods, effectively reduces IO processing latency and queue cache depth, avoids the waste of hardware resources under ordinary performance disks, and effectively reduces hardware costs.
[0007] To achieve the above objectives, one aspect of the present invention provides an NVMe acceleration system, comprising:
[0008] PCIe controller, which mounts several NVMe disks;
[0009] A memory controller is connected to a memory, which includes a first memory space and a second memory space.
[0010] The NVMe accelerator and CPU are connected to the PCIe controller and memory controller via on-chip interconnects, respectively. The CPU manages the first memory space, and the NVMe accelerator manages the second memory space.
[0011] According to one embodiment of the present invention, a management command queue and a management response queue are created in a first memory space. Each entry in the management command queue is a management command issued to the NVMe disk. The command is used to configure the NVMe disk and create an IO command queue for the NVMe disk. Each entry in the management response queue is a response to a management command. The response is written to the first memory space by the NVMe disk through the PCIe controller and read by the CPU.
[0012] According to one embodiment of the present invention, an IO command queue is created in a second memory space and the information of the IO command queue is stored in an NVMe disk. Each entry in the IO command queue is an IO command issued to the NVMe disk, and the IO command includes read commands and write commands.
[0013] According to one embodiment of the present invention, an IO completion queue is created in the internal cache of the NVMe accelerator, and the information of the IO completion queue is stored in the NVMe disk.
[0014] According to one embodiment of the present application, the NVMe acceleration system is configured such that the NVMe accelerator creates an IO command entry in the format of the read / write IO command of NVMe according to the received command and parameters of the read / write of the disk, and writes the IO command entry into the IO command queue, the NVMe accelerator accesses the PCIe controller through the interconnection bus, and writes the tail pointer of the IO command queue into the NVMe disk doorbell register through the write operation of the IO command doorbell register of the NVMe disk, after the NVMe disk receives the write operation of the doorbell register, the NVMe disk obtains the IO command from the IO command queue through the DMA operation and executes, when the IO command execution is completed, the NVMe disk organizes the IO completion entry, and writes the IO completion entry into the IO completion queue to notify the NVMe accelerator that the task has been executed.
[0015] According to one embodiment of the present application, the NVMe accelerator comprises:
[0016] a bus interface configured to receive the IO completion entry written by the NVMe disk;
[0017] a write detection module configured to monitor the preset signal of the bus interface, the preset signal comprising a write strobe signal and a write ready signal;
[0018] an IO completion queue management module configured to manage the head pointer and the tail pointer of the IO command queue.
[0019] Another aspect of the embodiment of the present application also provides a method for constructing an NVMe acceleration system, comprising:
[0020] mounting a plurality of NVMe disks on the PCIe controller;
[0021] connecting the memory to the memory controller, wherein the memory comprises a first memory space and a second memory space;
[0022] connecting the NVMe accelerator and the CPU to the PCIe controller and the memory controller through the on-chip interconnection respectively, wherein the CPU manages the first memory space, and the NVMe accelerator manages the second memory space.
[0023] According to one embodiment of the present application, the method further comprises:
[0024] creating a management command queue and a management response queue in the first memory space, each entry in the management command queue is a management command issued to the NVMe disk, the command is used for configuring the NVMe disk and creating the IO command queue of the NVMe disk, and each entry in the management response queue is a response to a management command, the response is written into the first memory space by the NVMe disk through the PCIe controller and read by the CPU.
[0025] According to one embodiment of the present application, creating a management command queue and a management response queue in the first memory space comprises:
[0026] The CPU opens up spaces in the first memory space for storing the management command queue and the management response queue respectively;
[0027] The PCIe controller writes the opened-up spaces in the system address into the NVMe disk through a write operation on the NVMe disk register.
[0028] According to one embodiment of the present application, creating an IO command queue in the second memory space and storing the information of the IO command queue into the NVMe disk, each entry in the IO command queue is an IO command issued to the NVMe disk, and the IO command includes a read command and a write command.
[0029] According to one embodiment of the present application, creating an IO command queue in the second memory space and storing the information of the IO command queue into the NVMe disk comprises:
[0030] The CPU opens up a space in the second memory space for storing the IO command queue;
[0031] The address of the opened-up space is written into the configuration register of the NVMe accelerator through the interconnection bus;
[0032] The CPU constructs a command for creating the IO command queue and puts the command into the management command queue;
[0033] The CPU notifies the NVMe disk of the new command by writing the management command doorbell register of the NVMe disk;
[0034] The NVMe disk obtains the command for creating the IO command queue from the location where the management command queue is located;
[0035] The NVMe disk parses the command for creating the IO command queue, and obtains and saves the information of the number of the IO command queue to be created, the location of the IO command queue in the memory, the depth of the IO command queue, and which IO completion queue the IO command queue is bound to.
[0036] According to one embodiment of the present application, creating an IO completion queue in the internal cache of the NVMe accelerator and storing the information of the IO completion queue into the NVMe disk.
[0037] According to one embodiment of the present application, creating an IO completion queue in the internal cache of the NVMe accelerator and storing the information of the IO completion queue into the NVMe disk comprises:
[0038] The CPU constructs a command of creating an IO completion queue and puts the command into a management command queue;
[0039] The CPU informs the NVMe disk of a new command by writing a management command doorbell register of the NVMe disk;
[0040] The NVMe disk obtains the command of creating the IO completion queue from a location where the management command queue is located;
[0041] The NVMe disk analyzes the command of creating the IO completion queue and obtains and saves a number of the IO completion queue to be created, a location of the IO completion queue in the memory and depth information of the IO queue.
[0042] Another aspect of the embodiment of the present application further provides a computer device, which comprises:
[0043] at least one processor; and
[0044] a memory, the memory storing computer instructions executable on the processor, the instructions being executed by the processor to implement the steps of any one of the above methods.
[0045] Another aspect of the embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps of any one of the above methods.
[0046] The NVMe acceleration system provided by the embodiment of the present application has the following beneficial technical effects: the PCIe controller is arranged, the PCIe controller is mounted with a plurality of NVMe disks; the memory controller is connected with the memory, the memory comprises a first memory space and a second memory space; the NVMe accelerator and the CPU are connected to the PCIe controller and the memory controller through on-chip interconnection respectively, the CPU manages the first memory space, and the NVMe accelerator manages the second memory space, so that the hardware can accelerate part of links in the NVMe protocol, high cost caused by the hardware to realize the complex NVMe protocol stack is avoided, the bottleneck of IO processing of the CPU is broken, high latency and access overhead caused by the traditional method are avoided, IO processing latency is effectively reduced and queue cache depth is reduced, hardware resource waste under the ordinary performance disk is avoided, and the hardware cost can be effectively reduced. BRIEF DESCRIPTION OF DRAWINGS
[0047] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other embodiments can be obtained based on these accompanying drawings without creative effort.
[0048] Figure 1 a schematic diagram of an NVMe acceleration system according to an embodiment of the present application;
[0049] Figure 2 a schematic diagram of memory management according to an embodiment of the present application;
[0050] Figure 3 a schematic diagram of an NVMe accelerator according to an embodiment of the present application;
[0051] Figure 4 a schematic flow chart of a method of constructing an NVMe acceleration system according to an embodiment of the present application;
[0052] Figure 5 a schematic diagram of a computer device according to an embodiment of the present application;
[0053] Figure 6 a schematic diagram of a computer readable storage medium according to an embodiment of the present application. DETAILED DESCRIPTION
[0054] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other embodiments can be obtained based on these accompanying drawings without creative effort.
[0055] Based on the above purpose, in a first aspect of the embodiments of the present application, an embodiment of an NVMe acceleration system is provided. Figure 1 A schematic diagram of the system is shown.
[0056] As shown in the above Figure 1 The system can include:
[0057] PCIe controller, PCIe controller mounts several NVMe disks. The NVMe disk is mounted under the PCIe controller, the PCIe controller has a root port, connects multiple NVMe disks through the PCIe switch, or the PCIe controller has multiple root ports, each root port is directly connected to an NVMe disk. Since the underlying NVMe protocol needs to use the PCIe protocol, the PCIe controller is mainly responsible for processing the PCIe protocol layer, and the PCIe controller is connected to the system through the interconnection bus, and the main purpose is to enable the CPU and the NVMe accelerator to interact with the NVMe disk through it.
[0058] Also includes a memory controller, the memory controller connects the memory, and the memory includes a first memory space and a second memory space. The memory can be a DRAM grain, and the memory controller is a DRAM controller, which manages the DRAM grain and provides memory space for the system. The management command queue, the management response queue, the IO command queue and the read-write data of the disk of the NVMe disk are placed in the DRAM, and the DRAM controller is connected to the system through the interconnection bus, and the main purpose is to enable the CPU, the NVMe accelerator and the PCIe controller to access. The NVMe accelerator designed in the application needs to be combined with the CPU to implement the NVMe protocol, so it is different from the traditional DRAM which is exclusively used by the CPU. It should be emphasized that although specific implementation methods such as memory and DRAM are used in this embodiment, the core is to use a storage space connected to the interconnection bus, which can be accessed by the CPU, the NVMe accelerator and the PCIe controller. The storage space can also be implemented by on-chip SRAM or other RAM.
[0059] Also includes NVMe accelerator and CPU, NVMe accelerator and CPU are connected to PCIe controller and memory controller through on-chip interconnection respectively, CPU manages first memory space, NVMe accelerator manages second memory space. As Figure 2As shown, the CPU and the NVMe accelerator jointly implement the NVMe protocol to access the NVMe disk, the CPU manages the first memory space, creates a management command queue and a management response queue in the first memory space, the two queues are used to implement the management of the NVMe disk, each entry in the management command queue is a management command issued to the disk, which is used to configure the NVMe disk and create an IO command queue of the disk, and each entry in the management response queue is a response to a management command, which is written into the first memory by the NVMe disk through the PCIe controller and read by the CPU. The NVMe accelerator manages the second memory space, creates an IO command queue in the second memory space, each entry of the IO command queue is an IO command issued to the disk, mainly read command and write command. An IO completion queue is created in the internal cache of the NVMe accelerator. The reason for this is that the reception event of a new response entry can be perceived through the monitoring of the read and write operations of the internal cache, instead of being perceived through constant polling as in the traditional method, which not only improves the detection speed, but also greatly reduces the access to the memory bus.
[0060] By using the technical solution of the application, the hardware can accelerate part of the links in the NVMe protocol, which not only avoids the high cost of implementing a complex NVMe protocol stack by hardware, but also breaks through the bottleneck of CPU IO processing, avoids the high latency and access overhead caused by the traditional method, effectively reduces the IO processing delay and reduces the queue cache depth, avoids the waste of hardware resources under the ordinary performance disk, and effectively reduces the hardware cost.
[0061] In a preferred embodiment of the application, a management command queue and a management response queue are created in the first memory space, each entry in the management command queue is a management command issued to the NVMe disk, the command is used to configure the NVMe disk and create an IO command queue of the NVMe disk, and each entry in the management response queue is a response to a management command, which is written into the first memory space by the NVMe disk through the PCIe controller and read by the CPU. The CPU first opens out spaces for storing the management command queue and the management response queue in the first memory space it manages, and then writes the addresses of the two spaces in the system to the NVMe disk register through the PCIe controller.
[0062] In a preferred embodiment of the present application, an IO command queue is created in the second memory space and information of the IO command queue is stored in the NVMe disk, each entry in the IO command queue is an IO command issued to the NVMe disk, and the IO command includes a read command and a write command. The CPU opens a space in the second memory space for storing the IO command queue, and then writes the configuration register of the NVMe accelerator through the interconnection bus to configure the location of the space to the NVMe accelerator. Generally, there can be multiple IO command queues, and the locations of the multiple IO command queues can not belong to continuous memory areas. The CPU constructs a command for creating an IO command queue, and then puts the command into the management command queue. The CPU notifies the NVMe disk of a new command by writing the management command doorbell register of the disk (the value written into the register is the tail pointer of the current management command queue), and the NVMe disk initiates a DMA operation to obtain the above-mentioned command for creating an IO command queue from the location of the management command queue in the memory. The NVMe disk parses the command for creating an IO command queue to obtain information such as the number of the IO command queue to be created, the location of the IO command queue in the memory, the depth of the IO command queue, and which IO completion queue the IO command queue is bound to, and the NVMe disk saves these information.
[0063] In a preferred embodiment of the present application, an IO completion queue is created in the internal cache of the NVMe accelerator, and information of the IO completion queue is stored in the NVMe disk. The CPU constructs a command for creating an IO completion queue, and then puts the command into the management command queue. The CPU notifies the NVMe disk of a new command by writing the management command doorbell register of the disk (the value written into the register is the tail pointer of the current management command queue), and the NVMe disk initiates a DMA operation to obtain the above-mentioned command for creating an IO completion queue from the location of the management command queue in the memory. The NVMe disk parses the command for creating an IO completion queue to obtain information such as the number of the IO completion queue to be created, the location of the IO completion queue in the memory (the IO completion queue is located in the internal cache of the NVMe accelerator), and other information such as the depth of the IO queue, and the NVMe disk saves these information.
[0064] In a preferred embodiment of the present application, the NVMe acceleration system is configured such that the NVMe accelerator creates an IO command entry according to the format of the NVMe read / write IO command based on the received command and parameters for reading and writing the disk, and writes the IO command entry into the IO command queue. The NVMe accelerator accesses the PCIe controller through the interconnection bus and performs a write operation on the IO command doorbell register of the NVMe disk to write the tail pointer of the IO command queue into the doorbell register of the NVMe disk. After receiving the write operation of the doorbell register, the NVMe disk obtains the IO command from the IO command queue through DMA operation and executes it. When the execution of the IO command is completed, the NVMe disk organizes an IO completion entry and writes it into the IO completion queue to notify the NVMe accelerator that the task has been completed. Other hardware modules in the system that need to access the NVMe disk send the NVMe disk read / write command and parameters to the NVMe accelerator. The NVMe accelerator creates an IO command entry according to the command parameters and in the format of the NVMe read / write IO command, and writes it into the memory where the IO command queue is located (as described above, the space is created by the CPU, and then the space address is notified to the NVMe accelerator). Then the NVMe accelerator accesses the PCIe controller through the interconnection bus and performs a write operation on the IO command doorbell register of the NVMe disk to write the tail pointer of the IO command queue into the doorbell register of the disk. After receiving the write operation of the doorbell register, the NVMe disk obtains the IO command from the IO command queue in the memory through DMA operation and executes it. When the command is completed, the disk organizes an IO completion entry and writes it into the IO completion queue to notify the NVMe accelerator that the task has been completed. As can be seen from the above process, the present application realizes the accelerated processing of the NVMe accelerator for the IO command, but does not need to intervene in the processing of complex disk management commands, memory allocation and other operations. The CPU takes over the management command to realize complex NVMe protocol management command processing, disk configuration, memory management and other operations, and avoids the performance bottleneck of the CPU processing a large number of IO commands.
[0065] In a preferred embodiment of the present application, the NVMe accelerator comprises:
[0066] The bus interface is configured to receive the IO completion entry written by the NVMe disk;
[0067] The write detection module is configured to monitor the preset signals of the bus interface, and the preset signals include the write enable signal and the write ready signal;
[0068] The IO completion queue management module is configured to manage the head pointer and the tail pointer of the IO command queue. For example, Figure 3As shown, the cache of IO completion queue is located inside the NVMe accelerator, when the disk organizes the IO completion entry, it writes the entry into the completion queue through DMA write operation, which goes through PCIe controller to on-chip interconnect, then writes to the bus interface of NVMe accelerator, and finally initiates the operation on the cache of IO completion queue through the bus interface. The disk DMA operation can finally write the entry into the accelerator because the CPU specifies the address of the IO completion queue as the address of the cache inside the NVMe accelerator when sending the IO completion queue creation command to the disk. The write monitoring module in the IO completion queue management module will monitor the write operation in real time, which is achieved by monitoring some key signals in the bus interface, such as write strobe, write ready and other signals (the names of these signals may be different according to the bus protocol), when a write operation is captured, the module will update the queue tail pointer register, and notify the IO completion entry parsing module of the event.
[0069] There are several advantages to placing the IO completion queue inside the NVMe accelerator, first, due to the separation of the management queue and the IO queue, the memory needs to be accessed by both the CPU and the accelerator, so the access operation is independent of the above two modules, and the NVMe accelerator cannot perceive it, so the most common method is to poll the memory at intervals to know if new data has been written, this polling operation will greatly waste the valuable bandwidth of the memory; second, the interval time of polling will inevitably bring delay, that is, a new completion entry cannot be perceived immediately after it is written, but needs to be processed after the next read, which undoubtedly increases the delay of command processing. Considering that the disk needs a certain time to process the IO command, in order to improve the processing efficiency of the command, the accelerator is not in a waiting state before a command is completed, but needs to continue to process other commands. That is, within a certain period of time, the accelerator has multiple unfinished commands being processed at the same time. To achieve this mechanism, the IO command context is mainly relied on, when an IO command is constructed and sent to the NVMe disk, the key information of the command will be saved in the context cache corresponding to the command entry number, when the IO completion entry is received, the number information of the IO command corresponding to the completion entry is first obtained from the IO completion entry, and then the corresponding entry is taken out from the context cache.
[0070] Meanwhile, the application does not place the context cache inside the accelerator, but in the memory. The advantage of this is that the performance of each disk is not the same, and important performance includes the execution delay of the IO command and the IO processing rate (IOPS). Since the context needs to be saved after a command is sent to the disk until the command processing is completed. During this period, the total number of pending commands that need to be received is the required cache size, that is, the number of context entries = IO command delay * IOPS. Therefore, for disks with different performance, the total number of caches required is uncertain, and placing it inside the accelerator will cause a performance bottleneck or waste of hardware resources. In the application, the NVMe accelerator records the execution time of each IO command through the context, and then calculates the average IO command delay, and then the CPU can adjust the cache size according to the required target IOPS. The adjustment method is to configure the cache size value to the NVMe accelerator through the register, and the NVMe accelerator ensures that it does not exceed the boundary when accessing the context cache according to the above value.
[0071] The system and data scheduling method designed by the application have the following advantages compared with the existing solutions:
[0072] 1. A mechanism of software and hardware cooperation is designed, which accelerates part of the links in the NVMe protocol by hardware, avoiding the high cost of implementing a complex NVMe protocol stack by hardware, and breaking through the bottleneck of CPU IO processing.
[0073] 2. The high latency and access overhead caused by the traditional method of completing the queue through interruption and polling are avoided, which not only effectively reduces the IO processing delay, but also reduces the queue cache depth and cost.
[0074] 3. The cache depth can be dynamically set according to the performance of different disks, which supports high-performance disks and avoids waste of hardware resources for ordinary performance disks, and can effectively reduce hardware cost.
[0075] It should be noted that those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the above-mentioned program can be stored in a computer readable storage medium. The program can include the processes of the above-mentioned embodiment methods when executed. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc. The above-mentioned computer program embodiments can achieve the same or similar effect as the corresponding any method embodiments.
[0076] In addition, the method disclosed by the embodiment of the present application can also be implemented as a computer program executed by a CPU, which can be stored in a computer readable storage medium. When the computer program is executed by the CPU, the above-mentioned functions defined in the method disclosed by the embodiment of the present application are performed.
[0077] Based on the above-mentioned purpose, a second aspect of the embodiment of the present application proposes a method for constructing an NVMe acceleration system, as shown in the following Figure 4 The method comprises the following steps:
[0078] S1, mounting a plurality of NVMe disks to a PCIe controller;
[0079] S2, connecting a memory to a memory controller, wherein the memory comprises a first memory space and a second memory space;
[0080] S3, connecting an NVMe accelerator and a CPU to the PCIe controller and the memory controller respectively through an on-chip interconnection, wherein the CPU manages the first memory space and the NVMe accelerator manages the second memory space.
[0081] In a preferred embodiment of the present application, the method further comprises:
[0082] In the first memory space, a management command queue and a management response queue are created, each entry in the management command queue is a management command issued to the NVMe disk, and the command is used to configure the NVMe disk and create an IO command queue of the NVMe disk, and each entry in the management response queue is a response to a management command, which is written into the first memory space by the NVMe disk through the PCIe controller and read by the CPU.
[0083] In a preferred embodiment of the present application, creating the management command queue and the management response queue in the first memory space comprises:
[0084] The CPU opens a space in the first memory space for storing the management command queue and the management response queue respectively;
[0085] The opened space is written into the NVMe disk through the PCIe controller to write the address of the space in the system into the NVMe disk.
[0086] In a preferred embodiment of the present application, an IO command queue is created in the second memory space and information of the IO command queue is stored into the NVMe disk, each entry in the IO command queue is an IO command issued to the NVMe disk, and the IO command comprises a read command and a write command.
[0087] In a preferred embodiment of the present application, creating the IO command queue in the second memory space and storing the information of the IO command queue into the NVMe disk comprises:
[0088] The CPU opens up a space in the second memory space for storing the IO command queue;
[0089] The address of the opened space is written into the configuration register of the NVMe accelerator through the interconnection bus;
[0090] The CPU constructs a command for creating the IO command queue and puts the command into the management command queue;
[0091] The CPU notifies the NVMe disk of the new command by writing the management command doorbell register of the NVMe disk;
[0092] The NVMe disk obtains the command for creating the IO command queue from the location where the management command queue is located;
[0093] The NVMe disk parses the command for creating the IO command queue and obtains and saves the information of the number of the IO command queue to be created, the location of the IO command queue in the memory, the depth of the IO command queue and which IO completion queue the IO command queue is bound to.
[0094] In a preferred embodiment of the present application, the IO completion queue is created in the internal cache of the NVMe accelerator and the information of the IO completion queue is stored into the NVMe disk.
[0095] In a preferred embodiment of the present application, creating the IO completion queue in the internal cache of the NVMe accelerator and storing the information of the IO completion queue into the NVMe disk comprises:
[0096] The CPU constructs a command for creating the IO completion queue and puts the command into the management command queue;
[0097] The CPU notifies the NVMe disk of the new command by writing the management command doorbell register of the NVMe disk;
[0098] The NVMe disk obtains the command for creating the IO completion queue from the location where the management command queue is located;
[0099] The NVMe disk parses the command for creating the IO completion queue and obtains and saves the information of the number of the IO completion queue to be created, the location of the IO completion queue in the memory and the depth of the IO queue.
[0100] To achieve the above object, a third aspect of the embodiments of the present application provides a computer device. Figure 5 An embodiment of the computer device provided by the present application is shown in the schematic diagram. As shown in the figure,Figure 5 As shown, the embodiments of the present application comprise an apparatus including at least one processor 21 and a memory 22 storing computer instructions 23 executable by the processor, which when executed by the processor implement the above method.
[0101] Based on the above purpose, a fourth aspect of the embodiments of the present application provides a computer readable storage medium. Figure 6 As shown is a schematic diagram of an embodiment of the computer readable storage medium provided by the present application. As shown, the computer readable storage medium 31 stores a computer program 32 which when executed by a processor performs the above method. Figure 6 As shown, the computer readable storage medium 31 stores a computer program 32 which when executed by a processor performs the above method.
[0102] In addition, the method disclosed by the embodiments of the present application can also be implemented as a computer program executed by a processor, which can be stored in a computer readable storage medium. When the computer program is executed by the processor, the above functions defined in the method disclosed by the embodiments of the present application are performed.
[0103] In addition, the above method steps and system units can also be implemented by using a controller and a computer readable storage medium for storing a computer program which enables the controller to implement the above steps or unit functions.
[0104] Those skilled in the art will also appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the embodiments of the present application.
[0105] In one or more exemplary designs, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or other wire-based, fiber-based, or wireless technologies, then the coaxial cable, fiber optic cable, twisted pair, DSL, or other wire-based, fiber-based, or wireless technologies are included in the definition of medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
[0106] The foregoing is a summary of the example embodiments disclosed herein, but it should be noted that various changes and modifications can be made without departing from the scope of the embodiments disclosed by the claims. The functions, steps and / or actions of the method claims described herein need not be performed in any particular order. Furthermore, although elements of the embodiments disclosed herein can be described or claimed in individual form, other embodiments can also include a plurality of those elements in combination.
[0107] It should be understood that, as used herein, "a" or "an" can mean one or more things unless context clearly indicates otherwise. It should also be understood that "and / or" as used herein means any and all possible combinations of one or more of the associated listed items.
[0108] The above-mentioned example embodiment numbers of the embodiments disclosed herein are merely for description, and do not represent the advantages or disadvantages of the embodiments.
[0109] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program instructing relevant hardware to complete, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.
[0110] Those skilled in the art shall understand that the above discussion of any embodiment is only exemplary, and is not intended to imply that the scope (including claims) of the embodiments of the present application is limited to these examples; the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes of different aspects of the embodiments of the present application as above. In order to be brief, they are not provided in details. Therefore, any omission, modification, equivalent replacement, improvement, etc. made in the spirit and principle of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.
Claims
1. An NVMe acceleration system, characterized by, The application relates to a PCIe controller, a memory controller, an NVMe accelerator and a CPU. The PCIe controller is connected to a plurality of NVMe disks. The memory controller is connected to a memory, and the memory comprises a first memory space and a second memory space. The NVMe accelerator and the CPU are connected to the PCIe controller and the memory controller through on-chip interconnection respectively, the CPU manages the first memory space, and the NVMe accelerator manages the second memory space. A management command queue and a management response queue are created in the first memory space, each entry in the management command queue is a management command issued to the NVMe disk, the command is used for configuring the NVMe disk and creating an IO command queue of the NVMe disk, and each entry in the management response queue is a response to a management command, the response is written into the first memory space by the NVMe disk through the PCIe controller and read by the CPU. An IO command queue is created in the second memory space, and information of the IO command queue is stored into the NVMe disk, each entry in the IO command queue is an IO command issued to the NVMe disk, and the IO command comprises a read command and a write command. An IO completion queue is created in an internal cache of the NVMe accelerator, and information of the IO completion queue is stored into the NVMe disk.
2. The system according to claim 1, characterized in that, The NVMe acceleration system is configured to create an IO command entry according to the received command and parameters of disk reading and writing in the format of the NVMe reading and writing IO command, and write the IO command entry into the IO command queue, the NVMe accelerator accesses the PCIe controller through the interconnection bus, and writes the tail pointer of the IO command queue into the NVMe disk doorbell register through a write operation on the IO command doorbell register of the NVMe disk, after the NVMe disk receives the write operation of the doorbell register, the IO command is obtained from the IO command queue through a DMA operation and executed, when the IO command execution is completed, the NVMe disk organizes an IO completion entry, and writes the IO completion entry into the IO completion queue to notify the NVMe accelerator that the task has been executed.
3. The system according to claim 1, characterized in that, The NVMe accelerator comprises: a bus interface configured to receive the IO completion entry written by the NVMe disk; a write detection module configured to monitor preset signals of the bus interface, the preset signals comprising a write enable signal and a write ready signal; an IO completion queue management module configured to manage a head pointer and a tail pointer of the IO command queue.
4. A method of building an NVMe acceleration system, the method comprising: The application further relates to a method for managing an NVMe acceleration system, and the method comprises the following steps: mounting a plurality of NVMe disks on a PCIe controller; connecting a memory to a memory controller, wherein the memory comprises a first memory space and a second memory space; connecting an NVMe accelerator and a CPU to the PCIe controller and the memory controller through on-chip interconnection respectively, wherein the CPU manages the first memory space, and the NVMe accelerator manages the second memory space; The management command queue and the management response queue are created in the first memory space, each entry in the management command queue is a management command issued to the NVMe disk, the command is used to configure the NVMe disk and create an IO command queue of the NVMe disk, and each entry in the management response queue is a response to a management command, the response is written into the first memory space by the NVMe disk through the PCIe controller and read by the CPU; The IO command queue is created in the second memory space, and information of the IO command queue is stored into the NVMe disk, each entry in the IO command queue is an IO command issued to the NVMe disk, and the IO command includes a read command and a write command; The IO completion queue is created in the internal cache of the NVMe accelerator, and information of the IO completion queue is stored into the NVMe disk.
5. The method according to claim 4, characterized in that, The management command queue and the management response queue are created in the first memory space, including: The CPU opens spaces for storing the management command queue and the management response queue in the first memory space respectively; The address of the opened spaces in the system is written into the NVMe disk through the write operation on the NVMe disk register by the PCIe controller.
6. The method according to claim 4, characterized in that, The IO command queue is created in the second memory space, and information of the IO command queue is stored into the NVMe disk, including: The CPU opens a space for storing the IO command queue in the second memory space; The address of the opened space is written into the configuration register of the NVMe accelerator through the interconnection bus; The CPU constructs a command for creating the IO command queue, and puts the command into the management command queue; The CPU notifies the NVMe disk of a new command by writing the management command doorbell register of the NVMe disk; The NVMe disk obtains the command for creating the IO command queue from the location of the management command queue; The NVMe disk parses the command for creating the IO command queue, and obtains and saves the information of the number of the IO command queue to be created, the location of the IO command queue in the memory, the depth of the IO command queue and which IO completion queue the IO command queue is bound to.
7. The method according to claim 4, characterized in that, The IO completion queue is created in the internal cache of the NVMe accelerator, and information of the IO completion queue is stored into the NVMe disk, including: The CPU constructs a command for creating the IO completion queue, and puts the command into the management command queue; The CPU notifies the NVMe disk of a new command by writing the management command doorbell register of the NVMe disk; The NVMe disk obtains the command for creating the IO completion queue from the location of the management command queue; The NVMe disk parses the command for creating the IO completion queue, and obtains and saves the information of the number of the IO completion queue to be created, the location of the IO completion queue in the memory and the depth of the IO queue.
8. A computer device, characterized by The computer program is executed by the processor to implement the steps of the method of any one of claims 4-7. The computer program is executed by the processor to implement the steps of the method of any one of claims 4-7. 9. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising:
Citation Information
Patent Citations
System and method for TCP offload for NVME over TCP-ip
CA3021969A1