A NVMe read request processing method and device
By analyzing PCIe TLP Prefix information and processing SQ/PRP read requests with high priority arbitration, the problem of insufficient PCIe bandwidth when concurrent access of NVMe disks is solved, and the data bandwidth utilization and performance between NVMe disks and NVMe HBA IOCs is fully utilized and performed. promote.
Patent Information
- Application Number
- CN202410750426.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-11
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-06-11
AI Technical Summary
When multiple NVMe disks access the host server concurrently, insufficient bandwidth of the upper PCIe leads to resource competition, resulting in the inability to read SQ/PRP information in time, resulting in the inability to fully optimize and utilize system performance.
By analyzing PCIe TLP Prefix information, determine whether the memory read request carries SQ or PRP identification information, send it to a high-priority queue, and use the arbitrator to conduct high-priority arbitration to send it to the host server to ensure that the SQ/PRP information is returned in a timely manner.
It realizes that the NVMe disk timely issues subsequent data read and write requests, avoids data outage, improves the overall performance of the group connected by the NVMe HBA IOC, and maintains the balance of data transmission without the need to update the host software.
Smart Images

Figure CN118606232B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of data storage, and in particular, to a method and device for processing NVMe read requests. Background Art
[0002] NVMe (NVM Express, non-volatile memory host controller interface specification), host adapter (HBA, HostBus Adapter), input output controller (IOC, I / O Controller) are widely used in computer network storage systems for communication between host servers and extended storage devices. The typical connection diagram is as follows: Figure 1 As shown, it includes: host server HS0; NVMe HBA IOC NHI0, which communicates with the host server (HSO) and NVMe disks (NVMe Disk) ND0, ND1, ND2...NDM based on the NVMe storage protocol, and the underlying interface for communicating with the host server and the NVMe disk is the PCIe interface; the upper PCIe interface controller UP0 is connected to communicate with the host; the lower PCIe interface controller DP0, DP1, DP2,...DPM is connected to communicate with the corresponding NVMe disk, as well as NVMe disks ND0, ND1, ND2,...NDM.
[0003] Generally, when multiple NVMe disks access the host server through NVMe HBA IOC simultaneously, the bandwidth of the downstream PCIe is greater than the bandwidth of the upstream PCIe, and there is resource competition for access to the upstream connection. Summary of the invention
[0004] The purpose of this application is to provide an NVMe read request processing method and device, aiming to solve the problem that SQ / PRP cannot be read in time when resource competition occurs in the upper PCIe, resulting in the inability to fully optimize the performance of the entire system.
[0005] According to a first aspect of the present application, a method for processing an NVMe read request is provided, comprising:
[0006] When receiving a memory read request sent from a PCIe NVMe disk, parse the memory read request PCIeTLP Prefix information to determine whether the memory read request carries SQ or PRP identification information;
[0007] If the memory read request carries SQ or PRP identification information, the current read request is sent to the SQ and PRP read request queues, otherwise the current read request is sent to the data read request queue, and the priority of the SQ and PRP read request queues is higher than that of the data read request queue;
[0008] The read requests in the SQ and PRP read request queues are sent to the read request sending queue of the host server through high priority arbitration, so that the host server returns the SQ or PRP information to the PCIe NVMe disk.
[0009] In an optional implementation manner, before parsing the PCIe TLP Prefix information of the memory read request to determine whether the memory read request carries SQ or PRP identification information, the method further includes:
[0010] When the PCIe NVMe disk sends a memory read request carrying an SQ or PRP request, the SQ or PRP identification enable information is set in the Local TLP PrefixType information.
[0011] In an optional implementation manner, before parsing the PCIe TLP Prefix information of the memory read request to determine whether the memory read request carries SQ or PRP identification information, the method further includes:
[0012] According to the configuration of the device capability configuration attribute Device Capability, the SQ or PRP identification enabling information is set in the Local TLP Prefix Type information.
[0013] In an optional implementation manner, sending the read requests in the SQ and PRP read request queues to the read request sending queue of the host server through high priority arbitration further includes:
[0014] The SQ and PRP read request queues and multiple data read request queues are connected to an arbitrator, and the arbitrator is used to perform Round-Robin arbitration between the multiple data read request queues, and a high priority is assigned to the SQ and PRP read request queues. When there are read requests carrying SQ or PRP identification information in the SQ and PRP read request queues, the corresponding read requests are sent to the read request sending queue in real time.
[0015] In an optional implementation, the NVMe read request processing method is executed by an NVMe HBA IOC.
[0016] According to a second aspect of the present application, an NVMe read request processing device is provided, including:
[0017] A parsing unit, configured to parse PCIe TLP Prefix information of a memory read request sent from a PCIe NVMe disk to determine whether the memory read request carries SQ or PRP identification information when receiving the memory read request.
[0018] a sending unit, configured to send the current read request to an SQ and PRP read request queue if the memory read request carries SQ or PRP identification information, and otherwise send the current read request to a data read request queue, wherein the SQ and PRP read request queues have a higher priority than the data read request queue;
[0019] An arbitration unit is used to send the read requests in the SQ and PRP read request queues to the read request sending queue of the host server through high priority arbitration, so that the host server returns the SQ or PRP information to the PCIeNVMe disk.
[0020] Compared with the related art, the technical solution of this application has the following advantages:
[0021] NVMe HBA IOC can identify SQ / PRP read requests and transmit them to the host with high priority arbitration, and read SQ Entry / PRP from the host and return it to the NVMe disk, so that the NVMe disk can issue subsequent data read and write requests in time and perform data access and transmission between the host. It also avoids the problem that the read and write bandwidth requirements of multiple disks are greater than the upper PCIe bandwidth but the data flow of NVMe HBA IOC reading and writing hosts is interrupted due to the failure to retrieve SQ / PRP of multiple disks in time. The data transmission balance between different disks will not be affected by the uneven SQ / PRP retrieval time delay between different disks, thereby fully optimizing the transmission data bandwidth between the NVMe disk and the NVMe HBA IOC, improving the overall performance of the disk group connected to the NVMe HBA IOC, and does not require the host software to be updated or modified, which is friendly to the current host software ecosystem.
[0022] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained through the structures and processes indicated in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. It is obvious that the drawings described below are certain embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0024] Figure 1 It is a schematic diagram of the NVMe HBA IOC connection relationship in the related technology.
[0025] Figure 2It is a schematic diagram of the data transmission process between NVMe Disk and host in the related technology.
[0026] Figure 3 It is a schematic diagram of the increase in SQ / PRP read request latency in the related art.
[0027] Figure 4 It is a flowchart of an NVMe read request processing method according to an exemplary embodiment of the present application.
[0028] Figure 5 It is a conceptual diagram of solving the problem of increased SQ / PRP read latency using a solution of PCIe and NVMe protocol extension according to an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0030] In the related art, when multiple NVMe disks access the host server through NVMe HBA IOC at the same time, the bandwidth of the downstream PCIe is greater than the bandwidth of the upstream PCIe, and there is resource competition for the access to the upstream. The brief communication process between NVMe Disk and the host server is as follows: Figure 2 As shown, including:
[0031] Step 1: After the host prepares the data storage area (Host Physical Memory) and the physical region page (PRP, Physical Region Page) (optional), the NVMe command (SQ Entry) is written to the submission queue SQ.
[0032] Step 2: The host writes Doorbell to the SQ Tail Doorbell register located in the NVMe Disk to indicate the new command.
[0033] Step 3: NVMe Disk reads the command (SQ Entry) from the host's SQ.
[0034] Step 4: NVMe Disk parses and executes the read command (SQ Entry):
[0035] Step 4A, NVMe Disk reads PRP from the host (optional).
[0036] Step 4B, NVMe Disk initiates data transmission to read data from the host.
[0037] Step 5: NVMe Disk writes CQ Entry to the host's completion queue CQ to indicate the most recently taken SQ Entry.
[0038] Step 6: After the command is executed, an MSI-X (Message Signaled Interrupt eXtended) interrupt is generated to notify the host that the command has been executed.
[0039] Step 7: The host processes the CQ Entry sent by the NVMe Disk.
[0040] Step 8: The host writes CQ Head Doorbell to the Completion Queue Head Doorbell register located in the NVMe Disk to indicate the processed CQ Entry.
[0041] From the above brief process, we can see that the command (SQ Entry) and PRP are actively moved from the host by the NVMe Disk. The subsequent data reading and writing are dependent on the acquisition of SQ Entry and PRP. That is, if the SQ Entry / PRP cannot be retrieved and parsed in advance or in time, the data reading and writing will be interrupted, resulting in a flow interruption.
[0042] When the NVMe disk connected to the NVMe HBA IOC accesses the host server concurrently, since the bandwidth of the lower PCIe is greater than the bandwidth of the upper PCIe, resource competition will occur for access to the host. In order to better balance the flow control of disk access to the host to achieve the best performance of the entire storage system, in engineering practice, when the upper PCIe sends a read request to the host, it will be sent according to the time interval (or slightly smaller) of the amount of data to be returned carried in the read request, that is, the next host read request will wait until the transmission time (or slightly smaller) required for the amount of data carried by the previous host read request arrives before it is sent to the host through the NVMeHBA IOC upper PCIe. Otherwise, a large number of concurrent read requests from the lower PCIe that have not been flow controlled and exceed the data bandwidth capacity that the upper PCIe can return are sent to the host, and a large number of read requests will be congested on the host side. On the one hand, it will lead to an increase in the delay of reading the host. On the other hand, the bigger problem is that when the disk received by the NVMe HBA IOC competes for access to the host resources, it is impossible to achieve balanced flow control of the received disk group, which results in the performance of the entire storage system not being optimal.
[0043] For NVMe disks, since the initiation of data read and write requests is dependent on the reading of SQ Entry / PRP, only when the SQ Entry / PRP on the host side is read into the disk and parsed, can the data read and write request corresponding to this SQEntry / PRP be initiated. For the scenario where the host writes to the disk concurrently, if the bandwidth of the read host request issued by the downstream disk is greater than the read data transmission bandwidth that can be provided by the upstream PCIe, there will be a situation where the read request competes with the upstream PCIe. Only the read request that obtains the arbitration right is sent to the host through the upstream PCIe, otherwise it needs to be in a waiting state until the arbitration authorization is obtained, resulting in an increase in the read data delay. The increase in delay is proportional to the sum of the bandwidths of the downstream PCIe read requests minus the increment of the upstream PCIe read bandwidth. For the current NVMe Over PCIe protocol, from the perspective of the PCIe protocol, all that is sent are memory read requests (Memory Read Request), and no distinction is made between ordinary data read requests and SQ Entry and PRP read requests. When the read requests of SQEntry / PRP and the read requests of ordinary data between different downstream PCIe devices compete for access to the upstream PCIe, they will be arbitrated as having the same priority, resulting in increased latency in the reading of SQ Entry and PRP, similar to the reading and writing of ordinary data. Figure 3 A conceptual diagram is given. Assume that the NVMe HBAIOC (NHI0) is connected to four NMVe disks (ND0, ND1, ND2, NDM), and the host (NS0) writes to the four NVMe disks concurrently (from the perspective of the NVMe protocol, the four disks are concurrently read by the host). Assume that the read request bandwidth of each disk is half of the bandwidth of the upper PCIe (UP0), and each disk's SQ and PRP read requests have accumulated read request data of the lower PCIe bandwidth × time period T to be sent.
[0044] from Figure 3It can be seen that since the sum of the read request data bandwidth of the lower PCIe is greater than the read data bandwidth of the upper PCIe, the read request transmission between the lower PCIe generates competition back pressure and is stored in the respective read request queues RdReq_Q0, RdReq_Q1, RdReq_Q2, and RdReq_QM, and waits for the Round-Robin arbiter RR_A0 of the lower PCIe read request queue to arbitrate and send it to the read request queue of the upper PCIe US_RdReq_Q, resulting in an increase in the delay in sending the read request. The sending delay of the SQ / PRP read request of each disk increases from T to 2T, that is, the delay of the SQ / PRP reading to the disk increases by T, which causes the issuance of the corresponding read / write host data access request in the SQ / PRP to generate a corresponding delay, that is, the disk's read and write access to this part of the host data will lag behind. Ultimately, even if the upstream PCIe is not congested with access contention and the SQ / PRP read request rhythm issued by the previous disk can ensure that the disk can achieve full-speed flow without contention on the upstream PCIe, this part of the data transmission will not be connected with the previous data access transmission and a brief interruption will occur. If this happens to multiple concurrent disks at the same time, the problem will spread to the upstream PCIe, that is, the upstream PCIe will also experience interruptions. In this way, on the one hand, the bandwidth demand of the multiple concurrently accessed disks for the upstream PCIe is greater than the maximum bandwidth that the upstream PCIe can provide. On the other hand, due to the increased read return delay of SQ Entry / PRP, the disk cannot issue host read and write requests in time, resulting in an interruption in the upstream PCIe, which ultimately reduces the performance of the entire system and cannot be fully optimized.
[0045] Based on the above analysis, the present application provides an NVMe read request processing method and device, which can carry identification information for distinguishing ordinary data read requests from SQ / PRP read requests in a PCIe Memory Read Request TLP (PCIe Transport Layer Packet) issued from an NVMe disk by simply extending the NVMe protocol and the PCIe protocol. When the NVMe HBA IOC receives the Memory Read Request TLP issued by the disk, it detects the identification information and performs corresponding transmission processing. For SQ / PRP type Memory Read Request, the host is accessed through the NVMe HBA IOC with high priority, and the SQ Entry (or PRP) is finally read from the host to the disk in a timely manner, so that the disk can promptly issue the corresponding read and write request for accessing the host in the SQEntry / PRP, avoiding the performance loss problem that the aforementioned SQ Entry and PRP cannot be retrieved in time, and there is no need to change the NVMe Driver on the host side, which is friendly to the system NVMe software ecosystem.
[0046] See also Figure 4 The NVMe read request processing method provided by the present application includes:
[0047] Step 401: When receiving a memory read request sent from a PCIe NVMe disk, parse the memory read request PCIe TLP Prefix information to determine whether the memory read request carries SQ or PRP identification information.
[0048] Exemplarily, the definition of the local TLP prefix type is first extended to carry the identification information of SQ / PRP. The definition of the extended Local TLP Prefix Type is shown in Table 1, where NVMe SQ / PRP is the newly extended Local TLP Prefix Type, and other types are the Local TLP Prefix Types already defined in the current PCIe Base Spefication.
[0049]
[0050]
[0051] Table 1
[0052] Secondly, the definition of NVMe Over PCIe Transport Specification is extended. When the PCIe NVMe disk sends a Memory Read Request of SQ / PRP, the Local TLP Prefix Type defined in Table 1 is used to carry the identification information of NVMeSQ / PRP.
[0053] In a further embodiment, in order to achieve backward compatibility, whether to use Local TLP Prefix Type to carry NVMe SQ / PRP identification information can be configured through the device's PCIe capability configuration attribute Device Capability.
[0054] Step 402: If the memory read request carries SQ or PRP identification information, the current read request is sent to the SQ and PRP read request queues, otherwise the current read request is sent to the data read request queue, and the SQ and PRP read request queues have a higher priority than the data read request queue.
[0055] Exemplarily, the SQ / PRP read request of the NVMe HBA IOC is processed with high priority transmission. Figure 5 As shown, NVMe HBA IOC NHI0 communicates with host server HS0 and NVMe disk (ND0, ND1, ND2...NDM) based on NVMe storage protocol, and the underlying interface of NHI0 to communicate with host server and NVMe disk is PCIe interface. Arbitrator PRR_A0 assigns high priority to Cmd_RdReq_Q, and other common data read request queues RdReq_Q0 / 1 / 2 / ...M have equal priority, and Round-Robin polling arbitration strategy is adopted between them. When the PCIe (DP0 / 1 / 2 / ...M) connected to NHI0 receives the MemoryRead Request TLP sent by the disk, it detects whether the Memory Read Request TLP contains Local TLP Prefix. If it contains and Local TLP Prefix Type = NVMe SQ / PRP, this request packet is delivered to the read request queue Cmd_RdReq_Q of SQ and PRP, otherwise it is delivered to RdReq_Q0 / 1 / 2 / ...M.
[0056] Step 403: Send the read requests in the SQ and PRP read request queues to the read request sending queue of the host server through high priority arbitration, so that the host server returns the SQ or PRP information to the PCIe NVMe disk.
[0057] Since Cmd_RdReq_Q has a high arbitration priority compared to the read request queue (RdReq_Q0 / 1 / 2…M) of ordinary data, Cmd_RdReq_Q will be sent to the read request sending queue US_RdReq_Q of the upper PCIe by the high priority arbitration, and finally read the SQ Entry / PRP from the host HS0 to the NVMe disk (ND0 / 1 / 2 / …M) in time. In a preferred embodiment, the corresponding read request in Cmd_RdReq_Q is sent to US_RdReq_Q in real time, so that the NVMe disk can issue the corresponding read and write data request in SQ Entry / PRP in time, solving the performance problem that SQEntry / PRP cannot be read and returned in time.
[0058] The above embodiment provides that the read request information of SQ / PRP is carried by Type in PCIe TLP Prefix. In other optional embodiments, as an alternative, the read request information of SQ / PRP can be carried directly by extending Type in PCIe TLP Header.
[0059] It can be seen that the NVMe read request processing method provided by this application has the following advantages compared with the related technologies:
[0060] NVMe HBA IOC can identify SQ / PRP read requests and transmit them to the host with high priority arbitration, and read SQ Entry / PRP from the host and return it to the NVMe disk, so that the NVMe disk can issue subsequent data read and write requests in time and perform data access and transmission between the host. It also avoids the problem that the read and write bandwidth requirements of multiple disks are greater than the upper PCIe bandwidth but the data flow of NVMe HBA IOC reading and writing to the host is interrupted due to the failure to retrieve SQ / PRP of multiple disks in time. The data transmission balance between different NVMe disks will not be affected by the uneven SQ / PRP retrieval time delay between different NVMe disks, thereby fully optimizing the transmission data bandwidth between the NVMe disk and NVMe HBA IOC, improving the overall performance of the disk group connected to the NVMe HBA IOC, and does not require the host software to be updated or modified, which is friendly to the current host software ecosystem.
[0061] Accordingly, the present application provides an NVMe read request processing device in a second aspect, including:
[0062] A parsing unit, configured to parse PCIe TLP Prefix information of a memory read request sent from a PCIe NVMe disk to determine whether the memory read request carries SQ or PRP identification information when receiving the memory read request.
[0063] a sending unit, configured to send the current read request to an SQ and PRP read request queue if the memory read request carries SQ or PRP identification information, and otherwise send the current read request to a data read request queue, wherein the SQ and PRP read request queues have a higher priority than the data read request queue;
[0064] An arbitration unit is used to send the read requests in the SQ and PRP read request queues to the read request sending queue of the host server through high priority arbitration, so that the host server returns the SQ or PRP information to the PCIeNVMe disk.
[0065] The above-mentioned device can be implemented by the NVMe read request processing method provided by the embodiment of the first aspect above. The specific implementation method can be found in the description in the embodiment of the first aspect, and will not be repeated here.
[0066] It is understood that the architectures, names and parameters described in the above embodiments are only examples. Those skilled in the art can also easily combine and adjust the structural features of the above embodiments according to the use requirements, and should not limit the concept of the present application to the specific details of the above examples.
[0067] Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that he or she may still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for processing an NVMe read request, characterized in that: include: When receiving a memory read request sent from a PCIe NVMe disk, parse the memory read request PCIe TLPPrefix information to determine whether the memory read request carries SQ or PRP identification information; If the memory read request carries SQ or PRP identification information, the current read request is sent to the SQ and PRP read request queues, otherwise the current read request is sent to the data read request queue, and the priority of the SQ and PRP read request queues is higher than that of the data read request queue; The read requests in the SQ and PRP read request queues are sent to the read request sending queue of the host server through high priority arbitration, so that the host server returns the SQ or PRP information to the PCIe NVMe disk.
2. The NVMe read request processing method according to claim 1, wherein: Before parsing the PCIe TLP Prefix information of the memory read request to determine whether the memory read request carries SQ or PRP identification information, the method further includes: When the PCIe NVMe disk sends a memory read request carrying an SQ or PRP request, the SQ or PRP identification enable information is set in the Local TLP Prefix Type information.
3. The NVMe read request processing method according to claim 2, wherein: Before parsing the PCIe TLP Prefix information of the memory read request to determine whether the memory read request carries SQ or PRP identification information, the method further includes: According to the configuration of the device capability configuration attribute Device Capability, the SQ or PRP identification enabling information is set in the Local TLP Prefix Type information.
4. The NVMe read request processing method according to claim 1, wherein: The sending of the read requests in the SQ and PRP read request queues to the read request sending queue of the host server through high priority arbitration further includes: The SQ and PRP read request queues and multiple data read request queues are connected to an arbitrator, and the arbitrator is used to perform Round-Robin arbitration between the multiple data read request queues, and a high priority is assigned to the SQ and PRP read request queues. When there are read requests carrying SQ or PRP identification information in the SQ and PRP read request queues, the corresponding read requests are sent to the read request sending queue in real time.
5. The NVMe read request processing method according to claim 1, wherein: The NVMe read request processing method is executed by the NVMe HBA IOC.
6. An NVMe read request processing device, characterized in that: include: A parsing unit, configured to parse PCIe TLP Prefix information of a memory read request sent from a PCIe NVMe disk to determine whether the memory read request carries SQ or PRP identification information when receiving the memory read request. a sending unit, configured to send the current read request to an SQ and PRP read request queue if the memory read request carries SQ or PRP identification information, and otherwise send the current read request to a data read request queue, wherein the SQ and PRP read request queues have a higher priority than the data read request queue; An arbitration unit is used to send the read requests in the SQ and PRP read request queues to the read request sending queue of the host server through high priority arbitration, so that the host server returns the SQ or PRP information to the PCIe NVMe disk.
7. The NVMe read request processing device according to claim 6, characterized in that: The parsing unit is further used for: When the PCIe NVMe disk sends a memory read request carrying an SQ or PRP request, the SQ or PRP identification enable information is set in the Local TLP Prefix Type information.
8. The NVMe read request processing method according to claim 7, characterized in that: The parsing unit is further used for: Before parsing the PCIe TLP Prefix information of the memory read request to determine whether the memory read request carries SQ or PRP identification information, the SQ or PRP identification enable information is set in the LocalTLP Prefix Type information according to the configuration of the device capability configuration attribute Device Capability.
9. The NVMe read request processing device according to claim 6, characterized in that: The arbitration unit is further used for: The SQ and PRP read request queues and multiple data read request queues are connected to an arbitrator, and the arbitrator is used to perform Round-Robin arbitration between the multiple data read request queues, and a high priority is assigned to the SQ and PRP read request queues. When there are read requests carrying SQ or PRP identification information in the SQ and PRP read request queues, the corresponding read requests are sent to the read request sending queue in real time.
10. The NVMe read request processing device according to claim 6, characterized in that: The NVMe read request processing device is located in the NVMe HBA IOC.
Citation Information
Patent Citations
Data access method, device and equipment
CN116301568A
System and Method for I / O Optimization in a Multi-Queued Environment
US20150134857A1
I / O driven data transfer in a data processing network
US20190129871A1
