Method of obtaining command processing time and storage device

Abstract

A method for obtaining command processing time and a storage device are provided. The provided method for processing I / O commands includes: in response to receiving an I / O command from a submission queue (SQ), recording the current value of a timestamp counter as a first value; generating a processing result for the I / O command, obtaining the current value of the timestamp counter as a second value, calculating the difference between the second value and the first value as the processing time of the I / O command; and updating an I / O command processing time interval count table based on the processing time of the I / O command; wherein the I / O command processing time interval count table is part of the log of the storage device.

Description

Technical Field

[0001] This application relates to storage technology, and more particularly to a method for obtaining the processing time of IO commands in a storage device, and a storage device thereof. Background Technology

[0002] Figure 1 A block diagram of the storage device is shown. Storage device 102 is coupled to a host computer to provide storage capabilities. The host computer and storage device 102 can be coupled in various ways, including but not limited to connections via SATA (Serial Advanced Technology Attachment), SCSI (Small Computer System Interface), SAS (Serial Attached SCSI), IDE (Integrated Drive Electronics), USB (Universal Serial Bus), PCIe (Peripheral Component Interconnect Express), NVMe (NVM Express), Ethernet, Fibre Channel, and wireless communication networks. The host computer can be an information processing device capable of communicating with the storage device via the above methods, such as a personal computer, tablet computer, server, laptop computer, network switch, router, cellular phone, or personal digital assistant. Storage device 102 includes interface 103, control unit 104, one or more NVM chips 105, and DRAM (Dynamic Random Access Memory) 110.

[0003] NAND flash memory, phase change memory, FeRAM (Ferroelectric RAM), MRAM (Magnetic Random Access Memory), RRAM (Resistive Random Access Memory), XPoint memory, etc. are common NVMs.

[0004] Interface 103 can be adapted to exchange data with the host via methods such as SATA, IDE, USB, PCIe, NVMe (NVM Express), SAS, Ethernet, and Fibre Channel.

[0005] The control unit 104 is used to control data transfer between the interface 103, the NVM chip 105, and the DRAM 110. It is also used for memory management, host logical address to flash physical address mapping, erase leveling, bad block management, etc. The control unit 104 can be implemented in various ways, including software, hardware, firmware, or a combination thereof. For example, the control unit 104 can be in the form of an FPGA (Field-programmable gate array), an ASIC (Application Specific Integrated Circuit), or a combination thereof. The control unit 104 may also include a processor or controller, in which software executes to manipulate the hardware of the control unit 104 to process I / O (Input / Output) commands. The control unit 104 can also be coupled to the DRAM 110 and can access the data in the DRAM 110. FTL tables and / or cached I / O command data can be stored in the DRAM.

[0006] The control unit 104 includes a flash interface controller (or media interface controller, flash channel controller), which is coupled to the NVM chip 105 and issues commands to the NVM chip 105 in accordance with the interface protocol of the NVM chip 105 to operate the NVM chip 105, and receives the command execution results output from the NVM chip 105. Known NVM chip interface protocols include "Toggle", "ONFI", etc.

[0007] Data is typically stored and retrieved in pages on storage media, while data is erased in blocks. A block (also called a physical block) contains multiple pages. A page on a storage medium (called a physical page) has a specified size, such as 17664 bytes. Physical pages can also have other sizes.

[0008] In storage devices, the FTL (Flash Translation Layer) is used to maintain mapping information from logical addresses to physical addresses. The FTL is implemented by, for example, software, firmware running in the CPU of the control unit, and / or hardware as part of an ASIC.

[0009] Figure 2 A schematic diagram illustrating existing methods for measuring I / O command processing time is shown.

[0010] The storage device is connected to the host. An application running on the host generates I / O commands to access the storage device. The host's operating system kernel (OS kernel) adds the I / O commands to the SQ (Submission Queue) according to, for example, the NVMe protocol. The storage device retrieves the I / O commands from the SQ, processes them, and adds the processing results to the CQ (Completion Queue). The OS kernel then retrieves the processing results from the CQ and delivers them to the application. The application, acting as storage device testing software, records the time (t1) when issuing an I / O command to the OS kernel and the time (t2) when receiving the processing results from the OS kernel. The processing time of the I / O command is then calculated by the difference between t2 and t1. Summary of the Invention

[0011] The processing time of I / O commands measured by the application is inaccurate. From the time the application issues an I / O command to the time it receives the result, there are several steps: the OS kernel processes the I / O command, the storage device processes it, and the I / O command is transmitted over the communication link from the host to the storage device. This results in a discrepancy between the measured I / O command processing time and the storage device's actual processing time. Therefore, it is desirable to obtain the accurate time for the storage device to process I / O commands. Furthermore, it is also desirable to obtain the time for the communication link to transmit the I / O command and the time for the OS kernel to process the I / O command.

[0012] According to a first aspect of this application, a method for processing a first I / O command according to the first aspect of this application is provided, comprising: in response to receiving an I / O command from a submission queue (SQ), recording the current value of a timestamp counter as a first value; generating a processing result of the I / O command, obtaining the current value of the timestamp counter as a second value, calculating the difference between the second value and the first value as the processing time of the I / O command; and updating an I / O command processing time interval count table according to the processing time of the I / O command; wherein the I / O command processing time interval count table is a part of the log of a storage device.

[0013] According to the first method for processing an I / O command according to the first aspect of this application, a second method for processing an I / O command according to the first aspect of this application is provided, further comprising: in response to a command to obtain storage device logs, providing an I / O command processing time counter table as a response to the command to obtain storage device logs.

[0014] According to the first or second method for processing I / O commands according to the first aspect of this application, a third method for processing I / O commands according to the first aspect of this application is provided, further comprising: if the difference between a first value and a second value is greater than a threshold, further recording the identifier of the I / O command, the difference, and / or the address accessed by the I / O command.

[0015] According to one of the first to third methods for processing I / O commands according to the first aspect of this application, a fourth method for processing I / O commands according to the first aspect of this application is provided, further comprising: adding the processing result of the I / O command to a completion queue (CQ), wherein the processing result of the I / O command includes the difference.

[0016] According to one of the first to fourth methods for processing IO commands according to the first aspect of this application, a fifth method for processing IO commands according to the first aspect of this application is provided, wherein the IO command processing time interval count table includes multiple entries, each entry of the IO command processing time interval count table representing a different time interval; a first entry of the IO command processing time interval count table is determined according to the time interval to which the difference belongs, and the count value of the first entry is incremented.

[0017] According to one of the first to fifth methods for processing I / O commands according to the first aspect of this application, a sixth method for processing I / O commands according to the first aspect of this application is provided, wherein a timestamp is generated using the host's current world time, and the timestamp is provided to a storage device via a timestamp setting command; the storage device sets the timestamp counter with the timestamp therein in response to receiving the timestamp setting command; the timestamp counter counts over time.

[0018] According to the sixth method for processing I / O commands according to the first aspect of this application, a seventh method for processing I / O commands according to the first aspect of this application is provided, further comprising: retrieving the I / O command from a submission queue (SQ) in response to a request, and recording a first current time in the I / O command; wherein, if the submission queue (SQ) is located in the host, the current time is represented by the world time of the host or the value of a timestamp counter maintained by the host; and if the submission queue (SQ) is located in the storage device, the current time is represented by the value of a counter maintained by the storage device.

[0019] According to the seventh method for processing I / O commands according to the first aspect of this application, an eighth method for processing I / O commands according to the first aspect of this application is provided, further comprising: in response to adding the processing result of the I / O command to a completion queue (CQ), recording a second current time in the processing result of the I / O command; wherein, if the completion queue (CQ) is located in the host, the value of a timestamp counter maintained by the host or the host represents the current time; and if the completion queue (CQ) is located in the storage device, the value of a counter maintained by the storage device represents the current time.

[0020] According to the eighth method for processing I / O commands according to the first aspect of this application, a ninth method for processing I / O commands according to the first aspect of this application is provided, further comprising: based on a second current time and a first current time carried in the processing result of the I / O command obtained from the completion queue (CQ), subtracting the first current time from the second current time to obtain a second difference, and subtracting the difference from the second difference to obtain the time for transmitting the I / O command and the processing result of the I / O command between the host and the storage device.

[0021] According to the ninth method for processing IO commands according to the first aspect of this application, a tenth method for processing IO commands according to the first aspect of this application is provided, wherein, when the value of the timestamp counter maintained by the host in the submission queue (SQ) represents the current time, the difference between the first current time carried in the processing result of the IO command obtained from the submission queue (SQ) and the first value is used as the time from retrieving the IO command from the submission queue (SQ) to receiving the IO command from the storage device.

[0022] According to the ninth or tenth method of processing IO commands according to the first aspect of this application, an eleventh method of processing IO commands according to the first aspect of this application is provided, wherein when the completion queue (CQ) is located in the host and the value of the timestamp counter maintained by the host represents the current time, the difference between the second current time carried in the processing result of the IO command obtained from the completion queue (CQ) and the second value is used as the time for transmitting the processing result of the IO command from the storage device to the completion queue (CQ).

[0023] According to one of the methods for processing I / O commands according to the seventh to eleventh aspects of the first aspect of this application, a method for processing I / O commands according to the first aspect of this application is provided in the twelfth aspect, further comprising: in response to adding the I / O command to the submission queue (SQ), adding a current time as a first moment in the I / O command; the storage device adding the first moment in the I / O command to the processing result of the I / O command; and using the difference between the time of the processing result of the I / O command obtained from the completion queue (CQ) and the first moment obtained from the processing result of the I / O command as the processing time of the I / O command.

[0024] According to a second aspect of this application, a storage system according to the second aspect of this application is provided, wherein the storage system includes a host and a storage device; the host issues an I / O command to the storage device; and the storage device executes a method for processing the I / O command according to the first to fifth aspects of this application to process one of the I / O commands.

[0025] According to a third aspect of this application, a first information processing apparatus according to the third aspect of this application is provided, including a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, when the processor executes the program, it implements one of the first to thirteenth methods for processing I / O commands according to the first aspect of this application. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0027] Figure 1 A block diagram showing a storage device using existing technology;

[0028] Figure 2 A schematic diagram illustrating existing methods for measuring I / O command processing time;

[0029] Figure 3A A schematic diagram illustrating a method for measuring I / O command processing time;

[0030] Figure 3B A swimlane diagram illustrating the measurement method for I / O command processing time;

[0031] Figure 4 A schematic diagram illustrating the measurement of I / O command processing time according to an embodiment of this application is shown;

[0032] Figure 5 A schematic diagram illustrating the measurement of I / O command processing time according to yet another embodiment of this application is shown;

[0033] Figure 6 A schematic diagram illustrating the measurement of I / O command processing time according to another embodiment of this application is shown; and

[0034] Figure 7 A schematic diagram illustrating the measurement of I / O command processing time according to another embodiment of this application is shown. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] Figure 3AA schematic diagram illustrating a method for measuring I / O command processing time is provided. Figure 3B A swimlane diagram illustrating the measurement method for I / O command processing time is shown.

[0037] See Figure 3A From the moment an application issues an I / O command to the moment it receives the processing result, several steps are involved. The application submits the I / O command through system calls provided by the OS kernel. Figure 3A (In, as indicated by (1)). See also Figure 3B The time point when the application submits an I / O command to the OS kernel is denoted as t1. The OS kernel processes the I / O command ( Figure 3A (2)). The OS kernel begins processing I / O commands at time t2. Figure 3B The OS kernel generates NVMe commands (hereinafter referred to as IO commands for simplicity) based on IO commands and adds them to SQ (System Queues). Figure 3A (3)), the current time is t3 ( Figure 3B The storage device obtains IO commands from the SQ (Storage Queues), and these commands are transmitted over a link connecting the host and the storage device, such as a PCIe link. Figure 3A In the middle, indicated by (4), at time t4 ( Figure 3B t5 is the start time of I / O command transmission on the PCIe link. Figure 3B The control unit of the storage device receives I / O commands from the host interface. Figure 3A (5)). The control unit issues one or more storage medium access commands to the NVM chip based on the IO commands. Figure 3A (6) Figure 3B For example, at times t6a and t6b, a storage medium access command is issued respectively. The NVM chip responds to the storage medium access command by providing the processing result to the control unit. Figure 3A ,(7)), time t7 ( Figure 3B The control unit receives the processing results of all storage medium access commands corresponding to the I / O commands. At time t8 ( Figure 3B The control unit transmits the processing results of the IO commands to the host via the host interface and PCIe link. Figure 3A (8)). t9 time ( Figure 3B The processing result of the IO command is written to CQ( Figure 3A (9)). t10 time ( Figure 3B The OS kernel obtains the I / O command processing results from CQ. Figure 3A (10)). Time t11 ( Figure 3B The application obtains the results of I / O command processing from the OS kernel. Figure 3A (11)).

[0038] The processing of I / O commands by the storage device begins with the control unit retrieving the I / O command from the storage queue (SQ) and ends with adding the processing result of the I / O command to the storage queue (CQ). Furthermore, in... Figure 3A In this example, SQ and CQ reside in host memory, and the time required to transmit IO commands or their processing results from SQ / CQ to the control unit over the PCIe link also needs to be considered.

[0039] Figure 4 A schematic diagram illustrating the measurement of I / O command processing time according to an embodiment of this application is shown.

[0040] When the control unit receives an IO command from the host interface (as indicated by (1)), it records the current time. The control unit maintains a start timetable for IO commands in process, which records the start time (the time the IO command is received from the host interface) of each IO command that the control unit is currently processing and has not yet completed. For example, the start timetable for IO commands in process shows four entries, recording the start time of IO command 410 as T412, IO command 420 as T422, IO command 430 as T432, and IO command 440 as T442. To make the recording of start times more accurate, the control unit adds a new entry to the start timetable for IO commands in process when it receives an IO command. The new entry records the identifier of the IO command and the current time as the start time. Figure 4 In the example, the start time of the IO command (e.g., T412) is the time when the control unit receives the IO command, not the time when the IO command leaves the SQ (not shown). After leaving the SQ, the IO command still needs to be transmitted via, for example, a PCIe link before reaching the control unit.

[0041] Optionally, some storage devices do not maintain the current time. For this purpose, a counter that continuously counts after power-on is used in the control unit, and its count value is taken as the current time. As another example, the control unit includes a timestamp register that records the number of milliseconds or microseconds that have elapsed since the control unit was powered on.

[0042] The control unit generates a storage media access command based on the IO command and provides it to the NVM chip (indicated by (2)). The NVM chip provides the processing result of the storage media access command to the control unit (indicated by (3)). The control unit generates an IO command processing result based on the processing result of the storage media access command and sends it to the CQ (not shown) in the host (indicated by (4)). Before sending the IO command (e.g., IO 420) processing result to the CQ, the control unit also obtains the current time (denoted as Tc) and the start time T422 of IO command 420 from the start time schedule of IO commands in processing, and obtains the processing time of IO command 420 (denoted as Tp) based on the current time and the start time T422. The entry recording the start time T422 of IO command 420 is deleted from the start time schedule of IO commands in processing. The control unit also maintains an IO command processing time interval count table. Each entry in the IO command processing time interval count table represents a specified IO command processing time interval and the number of IO commands processed within that time interval. For example, see Figure 4 The IO command processing time interval includes, for example, a 0-10µs (microsecond) interval, a 10-30µs (microsecond) interval, a 30-60µs (microsecond) interval, etc. The time interval division in the IO command processing time interval count table can be done in various ways to satisfy the observation of time intervals of different granularities or sizes. Optionally, the time intervals corresponding to each entry have the same size, for example, 10µs. As an example, if the processing time Tp of the control unit for IO command 420 is 50µs, then the control unit finds the entry covering the 50µs processing time in the IO command processing time interval count table. Figure 4 In the context of the entry (corresponding to the time interval of 30-60us), increment the count of that entry by 1 to reflect the addition of an IO command with a processing time in the 30-60us interval.

[0043] The count values ​​in the entries of the IO command processing time interval count table represent the number of IO commands processed within the corresponding time interval since the control unit was powered on or shipped from the factory. Optionally, the IO command processing time interval count table is also included when the control unit is powered off.

[0044] The I / O command processing time interval count table is generated as a storage device log by the control unit. The host obtains the I / O command processing time interval count table by accessing the storage device log. In applications, the host's applications can compare their own measured I / O command processing times (using existing counts) with the I / O command processing time interval count table obtained from the storage device log to further analyze the I / O command processing time and make adjustments or optimizations accordingly.

[0045] In alternative implementations, for example, the IO command is a write command, or the control unit provides a cache for read commands. The control device provides the IO command processing result to the CQ before generating a storage medium access based on the IO command, thereby... Figure 4 The instruction “(4)” occurs before “(2) / (3)”. The control unit updates the start timetable and IO command processing time interval count table for processing IO commands in the same way. Upon receiving an IO command, a new entry is added to the start timetable of the processing IO command, and the IO command and its start time are recorded. After the IO command processing is completed, before or simultaneously adding an IO command to the CQ, the corresponding entry is retrieved from the updated start timetable of the processing IO command based on the IO command's identifier, the processing time of the IO command in the control unit is calculated, and the count value of the time interval corresponding to the processing time of the IO command in the IO command processing time interval count table is updated. Understandably, the control unit can process hundreds or thousands of IO commands simultaneously, performing the same processing procedure on each IO command to obtain its processing time and update the IO command processing time interval count table. Optionally, the IO command identifier is indexed to the entry in the start timetable of the processing IO command, thus saving the time of searching for the entry in the start timetable of the processing IO command based on the IO command identifier. For example, a specified amount of storage space (accommodating a specified number of entries) is reserved for the start schedule of IO commands in process, the identifier of the IO command is hashed, and the result is used as the index of the entry in the start schedule of the IO commands in process.

[0046] Optionally, according to embodiments of this application, the control unit also records in the log the processing time of IO commands that exceed a specified threshold, along with their processing time. For example, after obtaining the processing time (length) of an IO command, if the processing time (length) is greater than, for example, 500 microseconds, the control unit, in addition to updating the IO command processing time interval count table, also records the IO command's identifier, processing time (length), optional accessed logical address, optional occurrence time, etc. The host can obtain such logs to identify IO commands with excessively long processing times and use them for further analysis of the causes.

[0047] Figure 5 A schematic diagram illustrating the measurement of I / O command processing time according to yet another embodiment of this application is shown.

[0048] In addition to obtaining the time it takes for the control unit to process I / O commands, it is also desirable to obtain the time it takes for the I / O commands to be transmitted over the PCIe link connecting the host and the storage device. The host typically has a clock and can obtain time information at any given moment (world time). However, storage devices typically do not have clocks and cannot obtain world time. Although storage devices can implement timing using counters, their time information is inconsistent with the host's world time.

[0049] According to embodiments of this application, before measuring the processing time of an I / O command, the host or its application sets a timestamp for the storage device. For example, the host sends a timestamp setting command to the storage device, such as the SetFeature command defined by the NVMe protocol. For example, the application converts the host's current UTC to the number of microseconds relative to a specified time base (e.g., January 1, 1970, 00:00), and sends this number of microseconds to the storage device via the SetFeature command. The storage device sets a timestamp counter based on the received number of microseconds, either from the set number or counting per microsecond. In this way, the host and the storage device obtain a common time represented by the number of microseconds relative to the time base.

[0050] To measure I / O command processing time, the application submits I / O commands to the OS kernel. Figure 5 In the middle (1), the OS kernel receives I / O commands ( Figure 5 In (2), the OS kernel adds IO commands to SQ ( Figure 5 (3)). According to an embodiment of this application, when the OS adds an IO command to the SQ, the current time (denoted as t3) is recorded in the added IO command. The current time is expressed in microseconds, for example, the current world time relative to a specified time base. The IO command in the SQ is transmitted to the storage device via a PCIe link ( Figure 5 In (4), the time it takes to retrieve an IO command from the SQ is recorded as t4. Time t4 is also expressed in microseconds, for example, the current world time relative to a specified time base. Time t4 is also recorded in the IO command and sent to the storage device along with it. Optionally, the DMA unit of the host or control unit adds time t4 to the IO command when initiating a transfer from the SQ. The control unit retrieves the IO command from the host interface ( Figure 5 The control unit obtains the current time (5) and records the current time as the start time of its I / O command processing (denoted as t5). The current time obtained by the control unit is the current value of its timestamp counter.

[0051] Figure 5 In this embodiment, each entry in the start time schedule of the processing IO command has a start time field. In response to receiving an IO command, the control unit creates an entry in the start time schedule of the processing IO command, records the identifier of the IO command in the entry, and uses the start time of the processing IO command as the start time field of the entry. For example, Figure 5The start timetable for I / O commands during processing includes four entries, recording the start time of I / O command 510 as T512, I / O command 520 as T522, I / O command 530 as T532, and I / O command 540 as T542. Optionally, the control unit does not need to maintain an additional start timetable for I / O commands during processing; instead, it records the start time of the processed I / O commands in the received I / O commands. The I / O commands also carry the time they are added to the SQ (t3) and / or the time they are retrieved from the SQ (t4).

[0052] The control unit generates storage media access commands based on the I / O commands and provides them to the NVM chip. Figure 5 In the middle, (6)), the NVM chip provides the processing result of the storage medium access command to the control unit ( Figure 5 In the middle, (7)). The control unit generates the processing result of the IO command based on the processing result of the storage medium access command (this moment is recorded as t8), and connects to the PCIe link through the host interface ( Figure 5 (8) provided to CQ ( Figure 5 (9), this moment is denoted as t9). Optionally, without generating storage medium access commands, the control unit can generate the processing results of IO commands, thereby Figure 5 The operation indicated by (8) occurs before the operation indicated by (6) and / or (7), or the operation indicated by (6) and / or (7) does not need to occur.

[0053] While the control unit generates the processing result (t8) of the IO command, it also retrieves the entry corresponding to the IO command from the start timetable of the IO commands in process, deletes the entry from the start timetable of the IO commands in process, and obtains the processing time (time length t8-t5) of the storage device for the IO command based on the current time obtained by, for example, a timestamp counter and the start time recorded in the entry. The IO command processing time interval count table is then updated based on the processing time of the IO command. Figure 5 (Not shown). Optionally, the control unit records the time (t5) at which it begins processing an IO command in the IO command itself, without needing to separately maintain the IO command start timetable. When generating the processing result of the IO command (t8), the value of time t5 is obtained from the IO command, and the processing time of the IO command by the storage device (time length t8-t5) is obtained. The host obtains information from the IO command processing time interval count table by accessing the storage device's log.

[0054] Optionally or further, the control unit also adds the start time (t5) of the recorded IO command processing and the time (t8) of generating the IO command processing result to the IO command processing result and provides them to the CQ. Optionally or further, the IO command processing result also includes the time when the IO command was added to the SQ. Understandably, in some cases, the IO command and the IO command processing result occupy the same storage space in the storage device; updating some fields in the IO command yields the IO command processing result. The time when the IO command was added to the SQ, carried by the IO command, already exists in the IO command processing result without additional operation.

[0055] Optionally or further, when the host adds the IO command processing result obtained from the PCIe link to the CQ, it also records a second current time (denoted as t9). The second current time is the number of microseconds in which the host's current world time is converted relative to a specified time base point when the host adds the IO command processing result to the CQ. Optionally, when the host or control unit's DMA writes all the IO command processing results to the CQ, it adds time t9 to it. Thus, the time (t3) when the host adds the IO command to the SQ and the time (t9) when it adds it to the CQ are both expressed in microseconds in which the world time when the IO command is added is relative to a specified time base point. Therefore, these two times can be compared, and their time difference represents the time length from when the IO command is added to the SQ to when the processing result of the IO command is added to the CQ.

[0056] Furthermore, according to the embodiments of this application, the time (t4) at which the host retrieves the IO command from the SQ can be compared with the start time (t5) of the IO command processing recorded by the control unit. The time difference represents the time it takes for the IO command to be transmitted from the host to the control unit on the PCIe link. Similarly, the time (t9) at which the host adds the IO command processing result to the CQ can be compared with the time (t8) at which the control unit generates the IO command processing result. The time difference represents the time it takes for the IO command processing result to be transmitted from the control unit to the host on the PCIe link. Thus, according to the embodiments of this application, in addition to the storage device obtaining the time length for processing each IO command and the number of IO commands in multiple intervals of the processing time length of the IO commands, the host obtains the time length between two operations based on the time recorded in the IO command processing results obtained from the CQ when it was added to the SQ and CQ, and also obtains the time when the IO command is transmitted from the host to the storage device, for example, via a PCIe link, based on the time when the IO command is retrieved from the SQ and the time when the control unit starts processing, and obtains the time when the IO command is transmitted from the storage device to the host, for example, via a PCIe link, based on the time when the IO command is added to the CQ (t9) and the time when the control unit finishes processing (t8).

[0057] Optionally, the control unit also obtains the time when the IO command is transmitted from the host to the storage device, for example, via a PCIe link, based on the time when the IO command is retrieved from the SQ (t4) and the time when it begins processing the IO command (t5), and records this time in a log accessible to the host.

[0058] For clarity, the current time relative to a specified time base point is called the timestamp time, to distinguish it from UTC. Based on... Figure 5 In this embodiment, the host synchronizes its timestamp with the storage device by setting the timestamp. However, the command to set the timestamp also takes time to be sent from the host to the storage device, which causes a difference between the host's timestamp and the storage device's timestamp, resulting in inaccurate vehicle time for the PCIe link.

[0059] Figure 6 A schematic diagram illustrating the measurement of I / O command processing time according to another embodiment of this application is shown.

[0060] according to Figure 6 In one embodiment, the host or its application sets a timestamp for the storage device before measuring the processing time of an IO command.

[0061] To measure I / O command processing time, the application submits I / O commands to the OS kernel. Figure 6 In the middle (1), the OS kernel receives I / O commands ( Figure 6 In (2), the OS kernel adds IO commands to SQ ( Figure 6 (3)). IO commands in SQ are transmitted to the storage device via the PCIe link ( Figure 6 In (4), the time when the IO command is retrieved from the SQ is recorded as t4. Time t4 is expressed in, for example, the host's world time and is recorded in the IO command. The control unit retrieves the IO command from the host interface ( Figure 6 The control unit (5) records the current time as the start time of its I / O command processing (denoted as t5). The current time obtained by the control unit is the current value of its timestamp counter. The control unit caches the I / O command and records the start time (t5) of processing the I / O command along with the cached I / O command.

[0062] The control unit generates the processing result of the IO command (this moment is recorded as t8), and connects it to the PCIe link via the host interface. Figure 6 (8) provided to CQ ( Figure 6 (9), this time is denoted as t9).

[0063] While generating the processing result (t8) of the IO command, the control unit also retrieves the time (t5) at which the control unit started processing the IO command from the cached IO commands. Based on the time (t8) and (t5) of generating the IO command processing result, the control unit calculates the time (t8-t5) used to process the IO command. The control unit updates the IO command processing time interval count table according to the processing time of the IO command. Optionally, the processing time of the IO command is also recorded in the IO command processing result and provided to the CQ.

[0064] When the host adds the IO command processing results obtained from the PCIe link to the CQ, it also records a second current time (denoted as t9), which is expressed by the host's world time.

[0065] According to embodiments of this application, the time t4 for retrieving the IO command from SQ and the time t9 for adding the IO command processing result to CQ are both expressed in the host's world time. The time t5 for the control unit to start processing the IO command and the time t8 for the control unit to generate the IO command processing result are both expressed by the timestamp counter value of the control unit. Time t9 is comparable to t4, and their difference represents the time length from retrieving the IO command from SQ to the IO command processing result being added to CQ. Time t8 is comparable to time t5, representing the time length for the control unit to process the IO command. However, t4 and t5 are not comparable, and t8 and t9 are not comparable.

[0066] Furthermore, (t9-t4)-(t8-t5) represents the transmission time of the IO command on the PCIe link, including the time it takes for the host's SQ to send the IO command to the storage device on the PCIe link, and the time it takes for the storage device to send the IO command processing result back to the host's CQ on the PCIe link. The CQ's IO command processing result carries the values ​​of t4 and t9, as well as the length of time the storage device processed the IO command (i.e., the value of t8-t5). Thus, the host application can know the length of time the storage device processed the IO command, and the transmission time of the IO command on the PCIe link.

[0067] As an example, although t4 and t5 are not comparable, and t8 and t9 are not comparable, the result of t5-t4 or t9-t8 can be used to estimate the time of IO command transmission on the (uplink or downlink) PCIe link.

[0068] Figure 7 A schematic diagram illustrating the measurement of I / O command processing time according to another embodiment of this application is shown.

[0069] exist Figure 7In this example, the Storage Queue (SQ) is built into the memory of the storage device, for example, within the Controller Memory Buffer (CMB) provided by the storage device according to the NVMe protocol. The Controller Memory Buffer (CQ) is built into the host's memory. The host writes I / O commands to the SQ located on the storage device and retrieves the processing results of the I / O commands from the CQ located on the host. Optionally, the CQ may also reside in the memory of the storage device.

[0070] Optionally, the host or its application sets a timestamp for the storage device before measuring the processing time of the IO command.

[0071] To measure I / O command processing time, the application submits I / O commands to the OS kernel. Figure 7 In the middle (1), the OS kernel receives I / O commands ( Figure 7 In (2), the OS kernel adds IO commands to the SQ located in the storage device via the PCIe link. Figure 7 In (3), a timestamp representing the current time t3 is added to the IO command. Since SQ is in the storage device, the timestamp representing time t3 is represented by the value of the timestamp counter. Control Unit

[0072] The time when the I / O command is retrieved in SQ is recorded as t4. Time t4 is expressed by the value of a timestamp counter and is also recorded in the I / O command. This time also serves as the time when the control unit begins processing the I / O command.

[0073] The control unit generates the processing result of the IO command (this moment is recorded as t5), and connects it to the PCIe link via the host interface. Figure 7 (5) provided to CQ ( Figure 7 In (6), this moment is denoted as t6). Time t5 is represented by, for example, the value of a timestamp counter, while time t6 is represented by, for example, the host's world time.

[0074] While generating the processing result of the IO command (time t5), the control unit also retrieves the time (t4) when the control unit started processing the IO command from the cached IO commands. Based on the time (t5) of generating the IO command processing result and the time (t4), the control unit calculates the time length (t5-t4) used to process the IO command. The control unit updates the IO command processing time interval count table according to the processing time length. Optionally, the processing time length of the IO command is also recorded in the IO command processing result and provided to the CQ.

[0075] When the host adds the IO command processing results obtained from the PCIe link to the CQ, it also records the current time (denoted as t6), which is expressed in the host's world time.

[0076] According to an embodiment of this application, the difference between time t5 and t4 represents the time length for the control unit to process the IO command. Time t6-t3 represents the time length from retrieving the IO command from SQ to adding the processing result of the IO command to CQ.

[0077] according to Figure 7 In this embodiment, the PCIe downlink transmission time from the host to the storage device is handled by the host's OS kernel, while the PCIe uplink transmission time from the storage device to the host is handled by the storage device.

[0078] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application. Clearly, those skilled in the art can make various alterations and variations to this application without departing from its spirit and scope. Thus, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for processing I / O commands, comprising: In response to receiving an IO command from the Submission Queue (SQ), the current value of the timestamp counter is recorded as the first value; wherein, the first value is: the control unit obtains the IO command from the host interface and records the current time as the start time of its processing of the IO command; The processing result of the IO command is generated, the current value of the timestamp counter is obtained as the second value, and the difference between the second value and the first value is calculated as the processing time of the IO command, wherein the processing time is the time for the control unit to process the IO command; the second value is the moment when the control unit generates the processing result of the IO command. Based on the processing time of the IO commands, update the IO command processing time interval count table. Each entry in the IO command processing time interval count table represents a specified IO command processing time interval and the number of IO commands processed within that time interval. The IO command processing time interval count table is part of the storage device's log.

2. The method according to claim 1, further comprising: In response to a command to retrieve storage device logs, the IO command processing time counter table is provided as a response to the command to retrieve storage device logs.

3. The method according to claim 1 or 2, further comprising: If the difference between the first value and the second value is greater than the threshold, the identifier of the IO command, the difference, and / or the address accessed by the IO command are also recorded.

4. The method according to claim 1, further comprising: The processing result of the IO command is added to the completion queue (CQ), wherein the processing result of the IO command includes the difference.

5. The method according to claim 1, wherein The IO command processing time interval count table includes multiple entries, each of which represents a different time interval. Based on the time interval to which the difference belongs, determine the first entry of the IO command processing time interval count table, and increment the count value of the first entry.

6. The method according to claim 1, wherein A timestamp is generated using the host's current world time, and the timestamp is provided to the storage device via a timestamp setting command; The storage device responds to receiving a timestamp setting command by setting the timestamp counter with the timestamp therein; The timestamp counter keeps ticking over time.

7. The method according to claim 6, further comprising: In response to a request, the IO command is retrieved from the submission queue (SQ), and a first current time is recorded in the IO command; in When the Submission Queue (SQ) is located in the host, the current time is represented by the host's world time or the value of a timestamp counter maintained by the host; as well as When the submission queue (SQ) is located in the storage device, the value of the counter maintained by the storage device represents the current time.

8. The method according to claim 7, further comprising: In response to adding the processing result of the IO command to the completion queue (CQ), a second current time is recorded in the processing result of the IO command; Where the completion queue (CQ) is located in the host, the current time is represented by the host's world time or the value of a timestamp counter maintained by the host; as well as When the completion queue (CQ) is located in the storage device, the value of the counter maintained by the storage device represents the current time.

9. The method according to claim 8, further comprising: Based on the second current time and the first current time carried in the processing result of the IO command obtained from the completion queue (CQ), the second current time is subtracted from the first current time to obtain a second difference, and the second difference is subtracted from the difference to obtain the time for the IO command to transmit the IO command and the processing result of the IO command between the host and the storage device.

10. The method of claim 9, wherein When the timestamp counter maintained by the host in the submission queue (SQ) represents the current time, the difference between the first current time carried in the processing result of the IO command obtained from the submission queue (SQ) and the first value is used as the time from retrieving the IO command from the submission queue (SQ) to receiving the IO command from the storage device.

11. The method of claim 9, wherein, when the value of the timestamp counter maintained by the host in the completion queue (CQ) represents the current time, the difference between the second current time carried in the processing result of the IO command obtained from the completion queue (CQ) and the second value is used as the time for transmitting the processing result of the IO command from the storage device to the completion queue (CQ).

12. The method according to claim 8, further comprising: In response to adding the IO command to the submission queue (SQ), the current time is added to the IO command as the first moment; The storage device adds the first moment in the IO command to the processing result of the IO command; The processing time of the IO command is the difference between the time of the processing result of the IO command obtained from the completion queue (CQ) and the first time obtained from the processing result of the IO command.

13. A storage system, wherein the storage system includes a host and storage devices; The host sends I / O commands to the storage device; The storage device executes the method of any one of claims 1-12 to process the I / O command.

14. An information processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for processing I / O commands as described in any one of claims 1-12.

Images