Linux kernel IO full link tracking positioning method and system

Abstract

This invention provides a Linux kernel I / O end-to-end tracing and localization method and system. The method includes: acquiring timestamps of I / O requests at multiple stages along the kernel I / O processing path using eBPF probes, and storing these timestamps in a bio information mapping table and a request information mapping table, respectively; upon completion of the I / O request, extracting the corresponding timestamps from the bio information mapping table and the request information mapping table; performing multi-stage time decomposition on the I / O request to obtain the time consumption of each stage; and locating the bottleneck stage causing the I / O request to slow down based on a comparison of at least one stage time consumption with a preset threshold. This application is dynamically implemented based on eBPF technology, requiring no modification to the kernel source code or system restart, achieving zero-intrusion, high-precision end-to-end observability.

Description

Technical Field

[0001] This invention belongs to the field of Linux kernel I / O processing, and in particular relates to a Linux kernel I / O full-link tracing and localization method and system. Background Technology

[0002] In the Linux operating system, a single I / O request from an upper-layer application requires collaborative processing through multiple layers, including the virtual file system, file system, block device layer, I / O scheduling layer, and driver layer. Existing I / O monitoring technologies have significant shortcomings: tools like iostat only reflect the overall load by reading aggregated statistics from the block device layer, failing to provide insight into the flow of I / O requests across different kernel layers; while blktrace can capture events at the block device layer, its tracing points do not cover the virtual file system layer, making it impossible to establish a complete I / O view from application to driver. More critically, these tools only provide the total latency of I / O operations. When performance issues occur, they cannot break down the total latency into fine-grained stages such as rate limiting, scheduling queuing, and driver processing, making it impossible for operations personnel to determine at which kernel layer the I / O bottleneck occurs, greatly increasing the difficulty of fault localization. Furthermore, some tracing tools require kernel modification or loading of kernel modules, introducing stability and security risks. Under high I / O pressure, the surge of monitoring events can exhaust system resources, triggering secondary disasters. Therefore, how to achieve full-link tracing of I / O from the application layer to the driver layer without making intrusive modifications to the kernel, and how to accurately decompose the total latency into the time consumption of multiple stages to locate bottlenecks, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0003] This application provides a Linux kernel I / O full-link tracing and localization method and system, which can automatically and directly determine whether the bottleneck is located in the driver hardware layer, scheduling layer or rate limiting layer, greatly improving the efficiency and accuracy of I / O performance diagnosis.

[0004] This application discloses a method for tracing and locating the entire Linux kernel I / O chain, including: On the kernel I / O processing path, the eBPF probe obtains the timestamps of I / O requests at multiple stages, and stores the timestamps in the bio information mapping table and the request information mapping table respectively. The bio information mapping table is a hash table with the bio structure address as the key, and the request information mapping table is a hash table with the request structure address as the key. In response to the completion of the IO request, the corresponding timestamp is extracted from the bio information mapping table and the request information mapping table, and the IO request is decomposed into multi-stage time consumption to obtain the time consumption of each stage. Based on the comparison of the time spent in at least one of the aforementioned stages with a preset threshold, the bottleneck stage causing the IO request to slow down is identified.

[0005] Optionally, the step of obtaining timestamps of IO requests at multiple stages through eBPF probes along the kernel IO processing path, and storing the timestamps in a bio information mapping table and a request information mapping table respectively, includes: During the IO submission phase, the first timestamp and process identifier are obtained, and the first timestamp and the process identifier are stored in the bio information mapping table with the bio structure address as the key. The second timestamp is obtained during the queue tag acquisition phase, and the third timestamp is obtained during the bio-to-request completion phase. The second timestamp and the third timestamp are stored in the request information mapping table using the request structure address as the key; A fourth timestamp is obtained during the request dispatch start processing phase, and a fifth timestamp is obtained during the request completion phase. The fourth and fifth timestamps are then updated in the request information mapping table.

[0006] Optionally, the step of performing multi-stage time decomposition on the IO request to obtain the time consumption of each stage includes: The rate limiting time is obtained based on the first timestamp and the second timestamp; The time taken to acquire the tag is obtained based on the second timestamp and the third timestamp; The scheduling time is obtained based on the third timestamp and the fourth timestamp; The driving time is obtained based on the fourth timestamp and the fifth timestamp.

[0007] Optionally, identifying the bottleneck stage causing the IO request to slow down based on a comparison of at least one stage's time consumption with a preset threshold includes: Compare the rate limiting time with the rate limiting time threshold; Compare the time taken to acquire the tag with the threshold for the time taken to acquire the tag; Compare the scheduling time with the scheduling time threshold; Compare the driving time with the driving time threshold; The bottleneck stage is determined based on the threshold items in the comparison results.

[0008] Optionally, before locating the bottleneck stage causing the IO request to slow down, the method further includes: Using the process identifier as an index, count the number of slow I / O location events generated by each process within a preset time window; When the number of times exceeds the preset reporting threshold, the current slow IO location event is discarded.

[0009] Optionally, before obtaining the timestamps of the IO request in multiple stages, the method further includes: Filter out the corresponding target devices based on the target device name or target device number specified by the user; Subsequent steps are performed only for IO requests sent to the target device.

[0010] Optionally, after identifying the bottleneck stage causing the IO request to slow down, the method further includes: Based on the inode information of the IO request, obtain and output the specific file path that caused the slow IO.

[0011] Optionally, locating the bottleneck stage that causes the IO request to slow down includes: The total I / O time is obtained based on the timestamps of the multiple stages; The step of locating the bottleneck stage based on the comparison result of at least one stage time with the preset threshold is only executed when the total IO time exceeds the preset slow IO time threshold.

[0012] Optionally, obtaining the first timestamp and process identifier during the IO submission phase includes: At the entry point of the blk_mq_make_request function, attach the first eBPF probe to obtain the first timestamp and the process identifier; The step of obtaining the second timestamp during the queue tag acquisition stage includes attaching a second eBPF probe at the return point of the blk_mq_get_tag function and obtaining the second timestamp. The process of obtaining the third timestamp during the bio-to-request completion phase includes attaching a third eBPF probe at the return point of the blk_mq_bio_to_request function to obtain the third timestamp. The step of obtaining the fourth timestamp during the request dispatch start processing phase includes attaching a fourth eBPF probe at the entry point of the blk_mq_run_request function to obtain the fourth timestamp; The step of obtaining the fifth timestamp during the request completion phase includes attaching a fifth eBPF probe at the entry point of the blk_mq_complete_request function to obtain the fifth timestamp.

[0013] This application also discloses a Linux kernel I / O full-link tracing and positioning system, the system comprising: The eBPF program module is used to obtain the timestamps of IO requests at multiple stages through eBPF probes on the kernel IO processing path, and store the timestamps in the bio information mapping table and the request information mapping table respectively. The bio information mapping table is a hash table with the bio structure address as the key, and the request information mapping table is a hash table with the request structure address as the key. The user-space program module is used to respond to the completion of the IO request by extracting the corresponding timestamp from the bio information mapping table and the request information mapping table, performing multi-stage time decomposition on the IO request, and obtaining the time consumption of each stage. The comparison and location module is used to locate the bottleneck stage that causes the IO request to slow down, based on the comparison result of at least one stage time with a preset threshold.

[0014] As can be seen from the above technical solution, during runtime, after an IO request is initiated, multiple eBPF probes accurately record timestamps at various moments, including IO submission, queue tag acquisition, bio to request conversion, request dispatch, and request completion. Specifically, the timestamp and process identifier of the IO submission phase are stored in a bio information mapping table with the bio address as the key. When the bio is converted to a request structure, the probe retrieves the aforementioned information from the bio information mapping table, combines it with the current timestamp, and stores it in a request information mapping table with the request address as the key. Subsequent dispatch and completion timestamps are directly updated in this request information mapping table. When the IO is completed, the system extracts the complete set of timestamps from both tables and performs time decomposition: the difference between the completion time and the dispatch time yields the driver time representing hardware and driver performance; the difference between the dispatch time and the bio to request conversion time yields the scheduling time reflecting scheduler efficiency; the difference between the bio to request conversion time and the queue tag acquisition time yields the tag acquisition time; and the difference between the queue tag acquisition time and the submission time yields the rate limiting time reflecting the IO congestion level. At this point, the previously chaotic total latency is clearly decomposed into four components with definite physical meanings. Finally, by comparing these components with their respective preset thresholds, the system can automatically and directly determine whether the bottleneck is located in the driver hardware layer, scheduling layer, or rate limiting layer, greatly improving the efficiency and accuracy of IO performance diagnosis. The entire process is dynamically implemented based on eBPF technology, without requiring modification of the kernel source code or a system restart, achieving zero-intrusion, high-precision end-to-end observability. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating a Linux kernel I / O full-link tracing and localization method in an embodiment of this application; Figure 2 This is a schematic diagram of the process of a Linux kernel I / O full-link tracing and localization method in this application embodiment, which obtains the timestamps of I / O requests at multiple stages on the kernel I / O processing path through eBPF probes and stores the timestamps in the bio information mapping table and the request information mapping table respectively. Figure 3 This is a flowchart illustrating the process of decomposing the time consumption of the IO request into multiple stages and obtaining the time consumption of each stage in a Linux kernel IO full-link tracing and localization method according to an embodiment of this application. Figure 4 This is a flowchart illustrating a Linux kernel I / O full-link tracing and localization method in an embodiment of this application, which locates the bottleneck stage that causes the I / O request to slow down based on the comparison result of the time consumption of at least one stage with a preset threshold. Figure 5 This is a flowchart illustrating the process before the bottleneck stage causing the IO request to slow down is located in a Linux kernel IO full-link tracing and localization method according to an embodiment of this application. Figure 6 This is a flowchart illustrating the process of a Linux kernel I / O full-link tracing and localization method in an embodiment of this application before obtaining the timestamps of the I / O request at multiple stages; Figure 7 This is a flowchart illustrating the process of locating the bottleneck stage that causes the IO request to slow down, according to a Linux kernel IO full-link tracing and localization method in this application embodiment. Figure 8 This is a flowchart illustrating the process of obtaining the first timestamp and process identifier during the IO submission phase in a Linux kernel IO full-link tracing and localization method according to an embodiment of this application. Figure 9 This is a schematic diagram of the structure of a Linux kernel I / O full-link tracing and positioning system in an embodiment of this application. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not limiting, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without such specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0018] This application relates to a Linux kernel I / O full-link tracing and localization method and system. To facilitate understanding of the technical solution of this application, typical solutions in the prior art and their existing problems are first described.

[0019] In this embodiment, the prior art is first described in detail. Taking an application server in a company's data center as an example, this server runs multiple business processes responsible for handling user read and write requests. One day, the operations and maintenance personnel found that the server response was slow, and users reported a significant increase in operation latency. The operations and maintenance personnel first used a common system monitoring tool to check the IO status. The tool showed that the average disk read and write speed was 100MB per second, the average wait time was 200 milliseconds, and the CPU IO wait rate reached 30%. These data indicated that the system had IO performance problems, but it was impossible to determine which specific stage the problem occurred in. The operations and maintenance personnel then used another block device layer tracing tool, which captured a large number of block device layer events, including request submission, dispatch, and completion. However, because this tool could not be correlated with the upper-layer application process and file system information, the operations and maintenance personnel could only know that the disk was processing a large number of IO requests, but could not know which process generated these requests, nor could they know which files these requests corresponded to. In addition, the tool could only give the total time for each IO request, and could not break down the total time into the time of different stages. The operations and maintenance personnel spent several hours trying various tools and methods, but ultimately failed to accurately pinpoint the exact location of the IO performance bottleneck, resulting in a prolonged disruption to business operations. This example fully demonstrates the shortcomings of existing technologies: they cannot achieve end-to-end full-link tracing, cannot perform fine-grained multi-stage time consumption analysis, and cannot accurately correlate processes with specific files. Consequently, when performance issues arise, operations and maintenance personnel struggle to quickly and accurately locate the root cause of the problem.

[0020] It should be noted that the examples of the existing technologies described above are only one of many application scenarios. In other similar scenarios, the existing technologies also suffer from the aforementioned shortcomings. For instance, in database servers, when queries become slow, existing technologies cannot determine at which stage of the database process's I / O requests is delayed, nor can they determine which database file's I / O operation is causing the performance problem. In storage servers, when storage access latency increases, existing technologies cannot determine whether the problem lies with the storage device itself, the I / O scheduling layer, or the rate limiting mechanism. These problems significantly complicate system operation and maintenance, as well as troubleshooting.

[0021] Based on this, embodiments of this application provide a Linux kernel I / O full-link tracing and location method, such as... Figure 1 As shown, it includes: S100: Obtain the timestamps of IO requests at multiple stages through eBPF probes on the kernel IO processing path, and store the timestamps in the bio information mapping table and the request information mapping table respectively. The bio information mapping table is a hash table with the bio structure address as the key, and the request information mapping table is a hash table with the request structure address as the key.

[0022] S200: In response to the completion of the IO request, extract the corresponding timestamp from the bio information mapping table and the request information mapping table, and perform multi-stage time decomposition on the IO request to obtain the time consumption of each stage.

[0023] S300: Based on the comparison result of the time consumed in at least one of the aforementioned stages with a preset threshold, locate the bottleneck stage that causes the IO request to slow down.

[0024] In this embodiment, eBPF is a technology provided by the Linux kernel that allows users to run sandbox programs in kernel space without modifying the kernel source code. eBPF programs are verified by the kernel verifier to ensure they will not cause kernel crashes and have high execution efficiency. eBPF supports various types of probes, including kprobe, uprobe, and tracepoint, which can be attached to the entry or return points of kernel functions to capture function execution information. In this application, the kprobe probe is mainly used to attach to key functions on the kernel I / O processing path to obtain the timestamps of I / O requests at various stages.

[0025] This application implements full-link tracing of Linux kernel I / O based on eBPF technology. The implementation must simultaneously meet the requirements of zero kernel intrusion, full-link timestamp concatenation, fine-grained time decomposition, and low resource consumption in extreme scenarios. eBPF programs run in a kernel sandbox environment, where stack space is strictly limited, making it impossible to store complete I / O request lifecycle data within a single probe function. Furthermore, different probe functions cannot directly share stack data; data exchange is only possible through mapping tables. The Linux kernel block device layer uses a blk-mq architecture. During I / O request processing, the bio structure is converted to a request structure. The lifecycles of these two structures are independent but partially overlap. The bio structure is reclaimed by the kernel after conversion and cannot be accessed in subsequent stages. To comprehensively record all timestamps from I / O request submission to completion, this application creates a bio information mapping table with the bio structure address as the key and a request information mapping table with the request structure address as the key. During the I / O submission phase, the initial timestamp and process identifier are stored in the bio information mapping table. When the blk_mq_get_tag function returns, a stable association has been established between the bio and request structures. At this point, the initial data is read from the bio information mapping table and stored together with the current timestamp in the request information mapping table. Timestamps from subsequent stages are updated in the request information mapping table, thus achieving complete timestamp transmission across the lifecycles of both structures. Based on the blk-mq architecture, this application obtains timestamps at five nodes: I / O submission, obtaining queue tags, bio to request completion, request dispatch starting processing, and request completion. The time consumption of the four stages is calculated by the difference between adjacent timestamps. The time consumption of each stage corresponds to the performance of different links in the I / O processing, directly reflecting the system's operating status in terms of I / O congestion, kernel processing, scheduling algorithms, and hardware drivers. In extreme I / O congestion scenarios such as disk failures, the system generates a large number of slow I / O events. Transmitting all these events to user space for processing would consume significant system resources. This application maintains a count of slow I / O events for each process within a fixed time window, indexed by process identifiers in the kernel space. When the count exceeds a preset threshold, the current event is discarded, controlling data transmission volume from the event's source and preventing monitoring actions from exacerbating system performance degradation. These technologies work together to achieve end-to-end I / O tracing from the application layer to the driver layer without modifying the kernel source code. This accurately decomposes the total I / O time and pinpoints performance bottlenecks, while ensuring system stability under various operating scenarios.

[0026] For example, the `bio` structure is a basic data structure in the Linux kernel used to describe block I / O requests. It contains information such as the starting sector, length, read / write direction, and device information of the I / O request. When an upper-layer application initiates an I / O request, after processing by the virtual file system and the file system layer, one or more `bio` structures are generated and then submitted to the block device layer for processing. The `request` structure is another data structure used by the block device layer to describe I / O requests; it is converted from the `bio` structure. At the block device layer, multiple `bio` structures may be merged into a single `request` structure to improve I / O processing efficiency. The `bio` and `request` structures have different lifecycles. The `bio` structure is created when the I / O is submitted and may be released after being converted to a `request` structure, while the `request` structure persists until the I / O request is completed.

[0027] It should be noted that the bio information map table and the request information map table are data structures provided by eBPF for sharing data between kernel space and user space. The map table is essentially a hash table that stores data in key-value pairs. In this application, the bio information map table uses the address of the bio structure as the key to store information such as the timestamp and process identifier of the IO request during the submission phase. The request information map table uses the address of the request structure as the key to store information such as the timestamps of the IO request in subsequent stages. Using two separate map tables can solve the problem of mismatched lifecycles between the bio and request structures, while also reducing the amount of data stored in each map table and improving data access efficiency.

[0028] Specifically, when an I / O request enters the kernel I / O processing path, the first timestamp and process identifier are obtained during the I / O submission phase, and this information is stored in the bio information mapping table. Then, when the I / O request enters the queue tag acquisition phase, the second timestamp is obtained. When the I / O request enters the bio-to-request-completion phase, the third timestamp is obtained. The second and third timestamps are stored in the request information mapping table. Next, when the I / O request enters the request dispatch start processing phase, the fourth timestamp is obtained. When the I / O request enters the request completion phase, the fifth timestamp is obtained. The fourth and fifth timestamps are updated in the request information mapping table. When the I / O request is completed, all timestamps are extracted from the bio information mapping table and the request information mapping table, and the time consumed in each phase is calculated. Then, the time consumed in each phase is compared with the corresponding preset threshold, and the bottleneck phase causing the I / O request to slow down is determined based on the comparison results.

[0029] For example, preset thresholds can be set according to the actual situation of the system. For instance, for a regular mechanical hard drive, the threshold for drive time can be set to 100 milliseconds. For a solid-state drive, the threshold for drive time can be set to 10 milliseconds. The threshold for rate limiting time can be set to 50 milliseconds, the threshold for tag acquisition time can be set to 1 millisecond, and the threshold for scheduling time can be set to 20 milliseconds. When the time of a certain stage exceeds the corresponding threshold, that stage can be considered the bottleneck stage causing the IO request to slow down.

[0030] It should be noted that the specific values ​​of the parameters or thresholds mentioned above in this application can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the purpose of this application can be achieved, and this application does not limit them in this regard.

[0031] Furthermore, those skilled in the art will understand that in practical applications, the division of the IO stages and the corresponding timestamp acquisition points can be increased or decreased as needed. For example, a timestamp acquisition point can be added to the virtual file system layer to obtain the time spent by the IO request in the virtual file system layer. More timestamp acquisition points can also be added to the driver layer to further decompose the time spent in the driver layer. These variations are all within the scope of protection of this application.

[0032] In alternative implementations, such as Figure 2 As shown, the step of obtaining timestamps of IO requests at multiple stages through eBPF probes along the kernel IO processing path, and storing these timestamps in the bio information mapping table and the request information mapping table respectively, includes: S110: During the IO submission phase, obtain the first timestamp and the process identifier, and store the first timestamp and the process identifier into the bio information mapping table with the bio structure address as the key.

[0033] S120: Obtain the second timestamp during the queue tag acquisition phase and the third timestamp during the bio to request completion phase.

[0034] S130: Store the second timestamp and the third timestamp into the request information mapping table using the request structure address as the key.

[0035] S140: Obtain the fourth timestamp during the request dispatch start processing phase, obtain the fifth timestamp during the request completion phase, and update the fourth timestamp and the fifth timestamp to the request information mapping table.

[0036] It's important to note that the IO submission phase refers to the stage where IO requests are submitted from the file system layer to the block device layer. During this phase, the kernel generates a `bio` structure and submits it to the block device layer's queue. The queue tag acquisition phase refers to the stage where the IO request waits to acquire a tag in the block device layer's queue. In the blk-mq architecture, each hardware queue has a certain number of tags used to identify the IO requests being processed. When an IO request enters the queue, it needs to wait to acquire an available tag before processing can continue. The `bio-to-request-complete` phase refers to the stage where the `bio` structure is converted into a `request` structure. During this phase, the kernel generates a `request` structure based on the contents of the `bio` structure and adds it to the scheduling queue. The request dispatch and processing start phase refers to the stage where the `request` structure is selected by the scheduler and dispatched to the driver layer for processing. The request completion phase refers to the stage where the driver layer completes the processing of the IO request and notifies the upper layer.

[0037] For example, the first timestamp is the time when the I / O request enters the block device layer, reflecting the processing time of the I / O request from the application layer to the block device layer. The second timestamp is the time when the I / O request obtains a queue tag, reflecting the time the I / O request waits in the queue to obtain a tag. The third timestamp is the time when the bio-to-request conversion is complete, reflecting the time the kernel converts the bio structure into a request structure. The fourth timestamp is the time when request dispatch begins, reflecting the time the request structure waits in the scheduling queue. The fifth timestamp is the time when the request completes, reflecting the time the driver layer processes the I / O request.

[0038] Specifically, the process identifier is a unique identifier used to identify the process that generated the I / O request. Through the process identifier, an I / O request can be associated with the process that generated it, thus identifying which process produced the slow I / O request. In this application, the process identifier is obtained during the I / O submission phase and stored in the bio information mapping table. When the bio structure is converted to a request structure, the process identifier is retrieved from the bio information mapping table and stored in the request information mapping table for subsequent use.

[0039] Furthermore, this application is not limited to this; in some embodiments, more process-related information, such as the process name, user ID, and group ID, can be obtained. This information can help operations and maintenance personnel gain a more comprehensive understanding of the source of IO requests, thereby enabling better troubleshooting.

[0040] In alternative implementations, such as Figure 3 As shown, the multi-stage time decomposition of the IO request to obtain the time consumption of each stage includes: S210: Calculate the rate limiting time based on the first timestamp and the second timestamp.

[0041] S220: Obtain the tag acquisition time based on the second timestamp and the third timestamp.

[0042] S230: The scheduling time is obtained based on the third timestamp and the fourth timestamp.

[0043] S240: The driving time is obtained based on the fourth timestamp and the fifth timestamp.

[0044] It's important to note that rate limiting latency refers to the time an I / O request waits in the block device layer queue to acquire a tag. In the blk-mq architecture, the number of tags in each hardware queue is limited. When the number of I / O requests in the queue exceeds the number of tags, new I / O requests must wait until a tag becomes available. Rate limiting latency reflects the degree of I / O congestion in the system; a longer latency indicates a higher I / O load and a large number of waiting I / O requests in the queue.

[0045] For example, assuming the first timestamp is 1,000,000 microseconds and the second timestamp is 10,000,500 microseconds, then the rate limiting time is 50 microseconds. This means that the IO request waits in the queue for 50 microseconds before obtaining the tag.

[0046] Tag acquisition time refers to the time from obtaining the tag from the I / O request to converting the bio structure into a request structure. This stage mainly involves internal kernel processing, including allocating the request structure and initializing its fields. Tag acquisition time reflects the kernel's processing efficiency. A long tag acquisition time may indicate high kernel CPU load or a problem with the kernel's processing logic.

[0047] Specifically, assuming the second timestamp is 1000050 microseconds and the third timestamp is 1000051 microseconds, then the time taken to obtain the tag is 1 microsecond. This means that the kernel takes 1 microsecond to convert the bio structure into a request structure.

[0048] Scheduling time refers to the time from when a request is converted from a bio to the point where it begins to be dispatched. During this stage, the request structure waits in the scheduling queue until it is selected by the scheduler and dispatched to the driver layer. Scheduling time reflects the efficiency of the I / O scheduler and the queue depth. If the scheduling time is long, it indicates that there are a large number of waiting request structures in the scheduling queue, which may be due to an inappropriate scheduling algorithm or an unreasonable queue depth setting.

[0049] For example, assuming the third timestamp is 1000051 microseconds and the fourth timestamp is 1000071 microseconds, the scheduling time would be 20 microseconds. This means that the request structure waited in the scheduling queue for 20 microseconds before being dispatched.

[0050] Driver latency refers to the time from the start of request dispatch to the completion of the request. During this stage, the driver layer interacts with the hardware device to complete the actual data read and write operations. Driver latency reflects the performance of the hardware device and the efficiency of the driver. If the driver latency is long, it may be due to poor performance of the hardware device itself or a problem with the driver.

[0051] Specifically, assuming the fourth timestamp is 1000071 microseconds and the fifth timestamp is 1000171 microseconds, then the driver's time consumption is 100 microseconds. This means that the driver layer took 100 microseconds to complete the processing of the IO request.

[0052] It should be noted that the specific values ​​of the parameters or thresholds mentioned above in this application can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the purpose of this application can be achieved, and this application does not limit them in this regard.

[0053] Furthermore, those skilled in the art will understand that the sum of the times consumed in the above four stages is the total time consumed by the I / O request. By breaking down the total time into the times consumed in these four stages, the processing time of the I / O request in each stage can be clearly known, thereby pinpointing the specific location of the performance bottleneck.

[0054] In alternative implementations, such as Figure 4 As shown, the step of locating the bottleneck stage causing the IO request to slow down based on the comparison result of at least one stage time with a preset threshold includes: S310: Compare the current limiting time with the current limiting time threshold.

[0055] S320: Compare the time taken to acquire the tag with the threshold value for the time taken to acquire the tag.

[0056] S330: Compare the scheduling time with the scheduling time threshold.

[0057] S340: Compare the driving time with the driving time threshold.

[0058] S350: Determine the bottleneck stage based on the threshold items in the comparison results.

[0059] It should be noted that the preset threshold is set in advance based on the actual situation and performance requirements of the system. Different systems, different hardware devices, and different application scenarios may require different preset thresholds. In practical applications, a reasonable threshold can be set by monitoring a normally operating system for a period of time, statistically analyzing the time distribution of each stage, and then using the statistical results.

[0060] For example, for a typical server using a mechanical hard drive, the rate limiting time threshold can be set to 50 milliseconds, the tag acquisition time threshold to 1 millisecond, the scheduling time threshold to 20 milliseconds, and the driver time threshold to 100 milliseconds. When the rate limiting time for a certain IO request exceeds 50 milliseconds, the rate limiting stage can be considered the bottleneck stage. When the tag acquisition time exceeds 1 millisecond, the tag acquisition stage can be considered the bottleneck stage. When the scheduling time exceeds 20 milliseconds, the scheduling stage can be considered the bottleneck stage. When the driver time exceeds 100 milliseconds, the driver stage can be considered the bottleneck stage.

[0061] Specifically, if the total time for an I / O request is 200 milliseconds, including 10 milliseconds for rate limiting, 0.5 milliseconds for tag acquisition, 15 milliseconds for scheduling, and 174.5 milliseconds for driver execution, then the driver execution time exceeds the preset 100 millisecond threshold. Therefore, the driver stage can be identified as the bottleneck causing the I / O request to slow down. At this point, operations and maintenance personnel can focus on checking the hardware devices and drivers to see if there are any issues such as disk failure or outdated driver versions.

[0062] Furthermore, in some cases, the time taken in multiple stages may simultaneously exceed the threshold. For example, the rate limiting time for an IO request might be 60 milliseconds, the driver time might be 120 milliseconds, and the times taken in other stages might be within the threshold. In this case, both the rate limiting stage and the driver stage become bottleneck stages. Operations personnel then need to resolve both the IO congestion issue and the hardware performance issue simultaneously.

[0063] It should be noted that the specific values ​​of the parameters or thresholds mentioned above in this application can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the purpose of this application can be achieved, and this application does not limit them in this regard.

[0064] This application is not limited to this. In some embodiments, the bottleneck stage can be determined based on the proportion of time spent in each stage to the total time spent. For example, if the time spent in a certain stage accounts for more than 80% of the total time spent, then that stage can be considered the main bottleneck stage. This method can avoid misjudgment caused by unreasonable threshold settings.

[0065] In alternative implementations, such as Figure 5As shown, before identifying the bottleneck stage that causes the IO request to slow down, the process also includes: S250: Using the process identifier as an index, count the number of slow IO location events generated by each process within a preset time window.

[0066] S260: When the number of times exceeds the preset reporting threshold, discard the current slow IO location event.

[0067] It should be noted that in a production environment, when severe I / O performance issues occur, such as disk failure or RAID controller malfunction, the system may generate a large number of slow I / O events. Without frequency control of these events, a large number of event reports can exhaust the system's CPU and memory resources, further exacerbating performance problems and potentially causing system crashes. Therefore, this application designs a dynamic frequency limiting mechanism based on process identifiers to control the number of slow I / O event reports.

[0068] For example, the preset time window can be set to 1 second, and the preset reporting threshold can be set to 10. That is, each process can report a maximum of 10 slow I / O events within 1 second. When a process generates more than 10 slow I / O events within 1 second, the excess events are discarded and not reported to user space. This effectively controls the number of reported events and avoids event storms. It should be noted that the specific values ​​of the parameters or thresholds mentioned above can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the inventive purpose of this application is achieved; this application does not impose any limitations on this.

[0069] Specifically, the frequency limiting mechanism is implemented as follows: First, a frequency limiting mapping table is maintained in the kernel space. This table uses the process identifier as the key and stores the count of slow I / O events generated by each process within the current time window and the start time of the time window. When a slow I / O event occurs, the count and the start time of the time window corresponding to the process are checked first. If the current time is within the time window and the count is less than the preset reporting threshold, the count is incremented by 1, and the event is allowed to be reported. If the current time is within the time window, but the count has reached the preset reporting threshold, the event is discarded. If the current time has exceeded the time window, the count is reset to 1, the start time of the time window is updated to the current time, and then the event is allowed to be reported.

[0070] Furthermore, those skilled in the art will understand that the preset time window and preset reporting threshold can be adjusted according to the actual situation of the system. For example, for some systems with high IO performance requirements, the preset reporting threshold can be appropriately increased to capture more slow IO events. For some resource-limited systems, the preset reporting threshold can be appropriately decreased to reduce system resource consumption.

[0071] This application is not limited to this; in some embodiments, other frequency limiting strategies may also be employed. For example, frequency limiting may be based on device identifiers, meaning each device may report a maximum of a certain number of slow I / O events within a preset time window. An adaptive frequency limiting strategy may also be used, dynamically adjusting the preset reporting threshold based on system load. When the system load is high, the preset reporting threshold is decreased; when the system load is low, the preset reporting threshold is increased.

[0072] In alternative implementations, such as Figure 6 As shown, before obtaining the timestamps of the IO request in multiple stages, the process also includes: S000: Filter out the corresponding target devices based on the target device name or target device number specified by the user.

[0073] S010: Perform subsequent steps only for IO requests sent to the target device.

[0074] It should be noted that in practical applications, a server typically connects to multiple storage devices, such as system disks, data disks, and backup disks. Different storage devices may be used for different purposes, and their performance requirements and load conditions may also differ. If IO tracking is performed on all storage devices, a large number of unnecessary events will be generated, increasing system resource consumption. Therefore, this application provides a device filtering function, allowing users to specify the target devices to be monitored and only track and analyze IO requests sent to these target devices.

[0075] For example, users can specify the target device name via command-line arguments, such as vda or vdb, or they can specify the target device number, such as 8:0 or 8:16. The device number consists of a major device number and a minor device number. The major device number identifies the device type, and the minor device number identifies different devices of the same type. When the user specifies the target device, the eBPF program in the kernel will check whether the device corresponding to the IO request is the target device before obtaining the timestamp of the IO request. If it is the target device, the subsequent steps are executed. If it is not the target device, it returns directly without any processing.

[0076] Specifically, the device filtering implementation process is as follows: First, in the user-space program, the target device name or target device number input by the user is parsed and converted into a unified device number format. Then, the target device number is passed to the eBPF program in the kernel. The eBPF program maintains a device filtering mapping table in kernel space, which stores all target device numbers that need to be monitored. When an I / O request enters the kernel I / O processing path, the eBPF program first obtains the device number corresponding to the I / O request from the bio structure, and then checks whether the device number exists in the device filtering mapping table. If it exists, it continues to obtain the timestamp of the I / O request. If it does not exist, it returns directly.

[0077] Furthermore, those skilled in the art will understand that users can specify multiple target devices and monitor multiple devices simultaneously. Alternatively, no target device can be specified, in which case all storage devices will be monitored. In some embodiments, more flexible device filtering methods can also be supported, such as filtering based on device type, capacity, mount point, etc.

[0078] In an optional implementation, after identifying the bottleneck stage that causes the IO request to slow down, the method further includes: S400: Based on the inode information of the IO request, obtain and output the specific file path that caused the slow IO.

[0079] It's important to note that an inode is a fundamental data structure in the Linux file system used to describe files. It contains metadata information such as file size, creation time, modification time, permissions, and the location of data blocks. Each file has a unique inode number, which can be used to locate the corresponding file. In this application, by associating IO requests with inode information, the specific file path causing slow IO can be obtained, thus helping operations personnel to more accurately pinpoint the problem.

[0080] For example, when an I / O request is generated, the bio structure contains the inode information corresponding to that I / O request. During the I / O submission phase, the eBPF program can obtain the inode number from the bio structure and store it in the bio information mapping table. When the I / O request is completed, the user-space program can obtain the inode number from the events reported by the eBPF program, and then find the corresponding file path based on the inode number by traversing the mount points of the file system.

[0081] Specifically, the process of obtaining the file path is as follows: First, the user-space program maintains a list of mount points, which contains information about all mount points in the system. When the user-space program receives a slow I / O event containing an inode number, it iterates through the mount point list. For each mount point, it uses a system call to obtain the correspondence between the inode numbers and file paths of all files under that mount point. Then, it searches for an entry in these correspondences that matches the inode number in the event, finding the corresponding file path. If a corresponding file path is found under a mount point, the iteration stops, and the file path is output. If no corresponding file path is found under any mount point, the inode number is output.

[0082] Furthermore, this application is not limited to this. In some embodiments, the file path can be directly obtained in the kernel space, and then the file path, along with other event information, can be reported to the user space. This method can improve the efficiency of obtaining the file path, but it increases the processing complexity and resource consumption in the kernel space.

[0083] In alternative implementations, such as Figure 7 As shown, locating the bottleneck stage that causes the IO request to slow down includes: S360: Calculate the total IO time based on the timestamps of the multiple stages.

[0084] S370: The step of locating the bottleneck stage based on the comparison result of at least one stage time with the preset threshold is executed only when the total IO time exceeds the preset slow IO time threshold.

[0085] It's important to note that not all I / O requests require bottleneck identification. Only when the total time of an I / O request exceeds a preset slow I / O time threshold is it considered a slow I / O request and requires further analysis and troubleshooting. This filters out most normal I / O requests, reducing system resource consumption and improving system efficiency.

[0086] For example, the preset slow I / O time threshold can be set to 100 milliseconds. That is, only when the total time of an I / O request exceeds 100 milliseconds will the I / O request undergo multi-stage time decomposition and bottleneck stage identification. I / O requests with a total time of less than 100 milliseconds are considered normal I / O requests and do not require processing.

[0087] Specifically, the total I / O time is calculated by subtracting the first timestamp from the fifth timestamp. After an I / O request is completed, the total I / O time is calculated first. If the total I / O time exceeds a preset slow I / O time threshold, subsequent steps are executed, including multi-stage time decomposition, frequency limit checks, and bottleneck stage location. If the total I / O time does not exceed the preset slow I / O time threshold, the entry corresponding to the I / O request is directly deleted from the bio information mapping table and the request information mapping table, releasing resources.

[0088] Furthermore, those skilled in the art will understand that the preset slow I / O time threshold can be adjusted according to the actual situation and performance requirements of the system. For example, for some systems with high I / O performance requirements, the slow I / O time threshold can be appropriately reduced to detect potential performance problems earlier. For some systems with low I / O performance requirements, the slow I / O time threshold can be appropriately increased to reduce unnecessary processing.

[0089] In alternative implementations, such as Figure 8 As shown, obtaining the first timestamp and process identifier during the IO submission phase includes: S111: Attach the first eBPF probe at the entry point of the blk_mq_make_request function to obtain the first timestamp and the process identifier.

[0090] S112: The step of obtaining the second timestamp during the queue tag acquisition stage includes attaching a second eBPF probe at the return point of the blk_mq_get_tag function and obtaining the second timestamp.

[0091] S113: Obtaining the third timestamp during the bio-to-request completion phase includes attaching a third eBPF probe at the return point of the blk_mq_bio_to_request function to obtain the third timestamp.

[0092] S114: Obtaining the fourth timestamp during the request dispatch start processing phase includes attaching a fourth eBPF probe at the entry point of the blk_mq_run_request function to obtain the fourth timestamp.

[0093] S115: Obtaining the fifth timestamp during the request completion phase includes attaching a fifth eBPF probe at the entry point of the blk_mq_complete_request function to obtain the fifth timestamp.

[0094] It's important to note that the functions mentioned above are all key functions in the Linux kernel's blk-mq architecture. The `blk_mq_make_request` function is the entry point for the block device layer to handle I / O requests; it is called when a `bio` structure is submitted to the block device layer. The `blk_mq_get_tag` function retrieves a queue tag for the I / O request; its return indicates that the I / O request has been tagged. The `blk_mq_bio_to_request` function converts a `bio` structure to a `request` structure; its return indicates that the conversion is complete. The `blk_mq_run_request` function dispatches the `request` structure to the driver layer for processing; its call indicates that request processing has begun. The `blk_mq_complete_request` function notifies the upper layer that the I / O request has completed; its call indicates that the I / O request has been processed.

[0095] For example, the first eBPF probe is attached to the entry point of the `blk_mq_make_request` function. When this function is called, the probe is triggered, obtains the current timestamp as the first timestamp, and simultaneously obtains the process identifier of the current process. Then, the first timestamp and the process identifier are stored in the bio information mapping table using the address of the bio structure as the key.

[0096] The second eBPF probe is attached at the return point of the `blk_mq_get_tag` function. When this function returns, the probe is triggered, and the current timestamp is obtained as the second timestamp. At this point, the `bio` structure has already obtained the queue tag and is associated with the `request` structure. Therefore, the address of the `request` structure can be obtained from the function's parameters, and then the second timestamp can be stored in the request information mapping table using the address of the `request` structure as the key.

[0097] The third eBPF probe is attached at the return point of the `blk_mq_bio_to_request` function. When this function returns, the probe is triggered, and the current timestamp is obtained as the third timestamp. Then, the third timestamp is updated in the corresponding `request` structure entry in the request information mapping table.

[0098] The fourth eBPF probe is attached at the entry point of the `blk_mq_run_request` function. When this function is called, the probe is triggered, and the current timestamp is obtained as the fourth timestamp. Then, the fourth timestamp is updated in the corresponding `request` structure entry in the request information mapping table.

[0099] The fifth eBPF probe is attached to the entry point of the `blk_mq_complete_request` function. When this function is called, the probe is triggered, obtaining the current timestamp as the fifth timestamp. Then, the fifth timestamp is updated in the corresponding `request` structure entry in the request information mapping table. At this point, the IO request is complete, and all timestamps can be extracted from the bio information mapping table and the request information mapping table for subsequent processing.

[0100] Specifically, the mounting and unmounting of eBPF probes is handled by user-space programs using the libbpf library. The user-space program first compiles the eBPF program, generating the object file. Then, it loads the object file into kernel space using the libbpf library and mounts the probes. When the user stops monitoring, the user-space program unmounts all probes and releases resources in kernel space.

[0101] Furthermore, those skilled in the art will understand that the names and parameters of the aforementioned functions may differ depending on the kernel version. In different kernel versions, it may be necessary to adjust the probe mounting location and parameter acquisition method. The scope of protection of this application is not limited to the specific function names and mounting locations mentioned above; any method that mounts probes at key nodes in the IO processing path, obtains timestamps, and performs multi-stage time decomposition falls within the scope of protection of this application.

[0102] This application also discloses a Linux kernel I / O end-to-end tracing and positioning system, such as... Figure 9 As shown, the system includes: eBPF program module 11 is used to obtain the timestamps of IO requests at multiple stages through eBPF probes on the kernel IO processing path, and store the timestamps in a bio information mapping table and a request information mapping table respectively. The bio information mapping table is a hash table with the bio structure address as the key, and the request information mapping table is a hash table with the request structure address as the key.

[0103] User-mode program module 12 is used to extract the corresponding timestamp from the bio information mapping table and the request information mapping table in response to the completion of the IO request, and to perform multi-stage time decomposition on the IO request to obtain the time consumption of each stage.

[0104] The comparison and positioning module 13 is used to locate the bottleneck stage that causes the IO request to slow down based on the comparison result of at least one stage time with a preset threshold.

[0105] First, the command-line arguments input by the user are parsed, including the target device name or device number, slow I / O time threshold, time threshold for each stage, preset time window, and preset reporting threshold. Then, the eBPF program module is compiled and loaded into kernel space, mounting all probes. Next, the event loop is entered, continuously reading event information from the BPF ring buffer. For each event, the corresponding file path is first obtained based on the inode number, and then the event information and file path are formatted and output to the terminal or saved to a file. When the user presses Ctrl+C to stop monitoring, the user-space program module unloads all probes, releases resources in kernel space, and then exits.

[0106] Specifically, the BPF ring buffer is a high-performance data transfer mechanism provided by eBPF for transferring data between kernel space and user space. Compared with traditional mapping tables, the BPF ring buffer has higher throughput and lower latency, making it suitable for transferring large amounts of event data. In this application, the eBPF program module writes slow I / O event information into the BPF ring buffer, and the user-space program module reads event information from the BPF ring buffer.

[0107] Now, drawing on previous examples of existing technologies, we will describe the specific implementation of this application. In that example, an application server in a company's data center experienced an IO performance problem, and the operations and maintenance personnel were unable to accurately locate the root cause of the problem using existing tools. Now, we will use the system provided in this application to troubleshoot the problem.

[0108] First, the operations and maintenance personnel run the user-space program of this application on the server, specifying the monitoring target device as vda, with a slow IO time threshold of 100 milliseconds, a rate limiting time threshold of 50 milliseconds, a tag acquisition time threshold of 1 millisecond, a scheduling time threshold of 20 milliseconds, a driver time threshold of 100 milliseconds, a preset time window of 1 second, and a preset reporting threshold of 10. The user-space program loads the eBPF program module into the kernel space and mounts all probes. Then, the system begins monitoring IO requests sent to the vda device. When an IO request enters the kernel IO processing path, each eBPF probe is triggered, obtains the corresponding timestamp, and stores it in the bio information mapping table and the request information mapping table. When the IO request is completed, the eBPF program module calculates the total IO time. If the total IO time exceeds 100 milliseconds, the time of each stage is calculated again, and a frequency limit check is performed. If the event is allowed to be reported, the event information is sent to the user-space program module. After receiving the event information, the user-space program module obtains the corresponding file path based on the inode number, and then formats and outputs the event information and file path.

[0109] Table 1 Output Event Information

[0110] As can be seen from the event information output in Table 1, the total time for the first event was 524.56 milliseconds, of which the driver took 524.55 milliseconds, accounting for 99.99% of the total time, exceeding the preset threshold of 100 milliseconds. Therefore, it can be determined that the driver stage is the bottleneck stage causing this IO request to slow down. This IO request was generated by process 1234 (mysqld), and the corresponding file path is / var / lib / mysql / data / test.ibd. This indicates that there was a delay in the read and write operations of the database file at the driver layer, which may be caused by disk failure or performance degradation.

[0111] The second event took a total of 600.21 milliseconds, with the driver taking 600.20 milliseconds, also exceeding the preset 100-millisecond threshold. This IO request was generated by process 5678 (redis), and the corresponding file path is / var / lib / redis / dump.rdb. This also indicates a performance issue in the driver layer.

[0112] The third event took 150.23 milliseconds in total, with scheduling accounting for 150.22 milliseconds, or 99.99% of the total time, exceeding the preset 20-millisecond threshold. Therefore, the scheduling phase can be identified as the bottleneck causing the slowdown of this IO request. This IO request was generated by process 9999 (dd), and the corresponding file path is / tmp / test.img. This indicates severe queue backlog in the IO scheduler, possibly due to an inappropriate scheduling algorithm or an unreasonable queue depth setting.

[0113] Based on this information, maintenance personnel can quickly pinpoint the root cause of the problem. For driver-level issues, they can check the disk's health status for bad sectors or other hardware failures, and try updating the driver. For scheduling-level issues, they can adjust the I / O scheduling algorithm, such as changing the MQ-deadline scheduling algorithm to the Kyber scheduling algorithm, or increasing the I / O queue depth. It should be noted that the specific values ​​of the parameters or thresholds mentioned above in this application can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the purpose of this invention is achieved, and this application does not impose any limitations on them.

[0114] By using the system provided in this application, operations and maintenance personnel can accurately locate the specific location of IO performance bottlenecks in a short time, greatly shortening the troubleshooting time and reducing the impact on business operations.

[0115] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A Linux kernel IO full link tracking and positioning method, characterized in that, The method includes: On the kernel I / O processing path, the eBPF probe obtains the timestamps of I / O requests at multiple stages, and stores the timestamps in the bio information mapping table and the request information mapping table respectively. The bio information mapping table is a hash table with the bio structure address as the key, and the request information mapping table is a hash table with the request structure address as the key. In response to the completion of the IO request, the corresponding timestamp is extracted from the bio information mapping table and the request information mapping table, and the IO request is decomposed into multi-stage time consumption to obtain the time consumption of each stage. Based on the comparison of the time spent in at least one of the aforementioned stages with a preset threshold, the bottleneck stage causing the IO request to slow down is identified.

2. The method according to claim 1, characterized in that, The step of obtaining timestamps of IO requests at multiple stages through eBPF probes along the kernel IO processing path and storing these timestamps in a bio information mapping table and a request information mapping table, respectively, includes: During the IO submission phase, the first timestamp and process identifier are obtained, and the first timestamp and the process identifier are stored in the bio information mapping table with the bio structure address as the key. The second timestamp is obtained during the queue tag acquisition phase, and the third timestamp is obtained during the bio-to-request completion phase. The second timestamp and the third timestamp are stored in the request information mapping table using the request structure address as the key; A fourth timestamp is obtained during the request dispatch start processing phase, and a fifth timestamp is obtained during the request completion phase. The fourth and fifth timestamps are then updated in the request information mapping table.

3. The method according to claim 2, characterized in that, The multi-stage time decomposition of the IO request to obtain the time consumption of each stage includes: The rate limiting time is obtained based on the first timestamp and the second timestamp; The time taken to acquire the tag is obtained based on the second timestamp and the third timestamp; The scheduling time is obtained based on the third timestamp and the fourth timestamp; The driving time is obtained based on the fourth timestamp and the fifth timestamp.

4. The method according to claim 3, characterized in that, The step of locating the bottleneck stage causing the IO request to slow down based on a comparison of the time consumed in at least one of the stages with a preset threshold includes: Compare the rate limiting time with the rate limiting time threshold; Compare the time taken to acquire the tag with the threshold for the time taken to acquire the tag; Compare the scheduling time with the scheduling time threshold; Compare the driving time with the driving time threshold; The bottleneck stage is determined based on the threshold items in the comparison results.

5. The method according to claim 1, characterized in that, Before identifying the bottleneck causing the slowdown in the IO request, the process also includes: Using the process identifier as an index, count the number of slow I / O location events generated by each process within a preset time window; When the number of times exceeds the preset reporting threshold, the current slow IO location event is discarded.

6. The method according to claim 1, characterized in that, Before obtaining the timestamps of the IO request across multiple stages, the following is also included: Filter out the corresponding target devices based on the target device name or target device number specified by the user; Subsequent steps are performed only for IO requests sent to the target device.

7. The method according to claim 1, characterized in that, After identifying the bottleneck causing the slowdown in the IO request, the process further includes: Based on the inode information of the IO request, obtain and output the specific file path that caused the slow IO.

8. The method according to claim 1, characterized in that, The process of locating the bottleneck stage that causes the IO request to slow down includes: The total I / O time is obtained based on the timestamps of the multiple stages; The step of locating the bottleneck stage based on the comparison result of at least one stage time with the preset threshold is only executed when the total IO time exceeds the preset slow IO time threshold.

9. The method according to claim 2, characterized in that, The step of obtaining the first timestamp and process identifier during the IO submission phase includes: At the entry point of the blk_mq_make_request function, attach the first eBPF probe to obtain the first timestamp and the process identifier; The step of obtaining the second timestamp during the queue tag acquisition stage includes attaching a second eBPF probe at the return point of the blk_mq_get_tag function and obtaining the second timestamp. The process of obtaining the third timestamp during the bio-to-request completion phase includes attaching a third eBPF probe at the return point of the blk_mq_bio_to_request function to obtain the third timestamp. The step of obtaining the fourth timestamp during the request dispatch start processing phase includes attaching a fourth eBPF probe at the entry point of the blk_mq_run_request function to obtain the fourth timestamp; The step of obtaining the fifth timestamp during the request completion phase includes attaching a fifth eBPF probe at the entry point of the blk_mq_complete_request function to obtain the fifth timestamp.

10. A Linux kernel I / O end-link tracing and positioning system, characterized in that, The system includes: The eBPF program module is used to obtain the timestamps of IO requests at multiple stages through eBPF probes on the kernel IO processing path, and store the timestamps in the bio information mapping table and the request information mapping table respectively. The bio information mapping table is a hash table with the bio structure address as the key, and the request information mapping table is a hash table with the request structure address as the key. The user-space program module is used to respond to the completion of the IO request by extracting the corresponding timestamp from the bio information mapping table and the request information mapping table, performing multi-stage time decomposition on the IO request, and obtaining the time consumption of each stage. The comparison and location module is used to locate the bottleneck stage that causes the IO request to slow down, based on the comparison result of at least one stage time with a preset threshold.

Images