A Linux kernel IO event processing exception detection method and system

Abstract

This application provides a method and system for detecting anomalies in Linux kernel I / O event handling. The method includes: collecting the I / O request dwell time without locks through percpu variables at preset I / O blocking nodes in the Linux kernel storage stack bio and request stages; creating file nodes in debugfs to summarize and output the time-consuming data in response to user-space read requests; and creating an I / O tracking structure for each I / O stage of each block device to manage the entry and exit of instances in the I / O request stage. It can also track the I / O context by registering hooks in the Device Mapper framework, and achieve asynchronous data interaction between the kernel and user space using a lock-free circular buffer and the io_uring interface. This application achieves layered observation of the entire I / O chain, low-overhead lock-free operation, and can locate abnormal I / O in seconds, balancing observation accuracy and business I / O performance, without affecting business operations.

Description

Technical Field

[0001] This application belongs to the field of Linux kernel technology, and in particular relates to a method and system for detecting abnormalities in Linux kernel I / O event handling. Background Technology

[0002] The Linux kernel storage stack encompasses multiple layers, including the file system, general block layer, and device drivers. An anomaly in any of these layers can trigger I / O performance issues. Quickly and accurately locating the root cause of slow I / O is a core requirement for system maintenance and kernel development. Among existing mainstream I / O monitoring tools, iostat can only read aggregated statistics from procfs and output average performance metrics over a period of time. It cannot distinguish the specific kernel stack layer where the I / O bottleneck is located, nor can it capture instantaneous I / O performance fluctuations, failing to match the actual I / O situation perceived by the business side. While blktrace can capture details of single I / O events, it relies on static kernel tracepoints, which introduce significant performance overhead when enabled, making it unsuitable for routine deployment in production environments. Existing technologies cannot simultaneously meet the core requirements of low-overhead, routine operation and accurate observation of the entire I / O chain, nor can they achieve real-time tracking of the entire lifecycle of a single I / O request, making it difficult to quickly delineate and locate I / O performance problems in production environments. Summary of the Invention

[0003] This application provides a method for detecting anomalies in Linux kernel I / O event handling, which can simultaneously meet the needs of low-overhead, normalized operation and accurate, layered observation of the entire I / O chain. This application also provides a system for detecting anomalies in Linux kernel I / O event handling.

[0004] This application discloses a method for detecting anomalies in Linux kernel I / O event handling, including: At the preset IO blocking node in the Linux kernel storage stack, the dwell time of each IO request at the corresponding IO blocking node is collected by the percpu variable. The preset IO blocking node is the execution node that causes IO request blocking during the bio phase and request phase of the Linux kernel storage stack. A preset file node is created in the kernel debugfs file system. When a read request for the file node is received from user space, the total resident time in the percpu variable corresponding to all CPU cores is determined, and the total resident time is output to user space through the file node. For each I / O stage of each block device in the Linux kernel, a corresponding I / O trace structure is created. When an I / O request enters the corresponding I / O stage, a structure instance based on the I / O trace structure is generated and added to the corresponding trace list. When the I / O request leaves the corresponding I / O stage, the instance is removed from the list. The I / O request processing is monitored in real time for abnormalities based on the total dwell time and the structure instance.

[0005] Optionally, the step of collecting the dwell time of each IO request at the corresponding IO blocking node through the percpu variable at the preset IO blocking node of the Linux kernel storage stack includes: Determine the CPU core number currently executing the I / O request, and obtain the memory address of the percpu variable corresponding to that CPU core; Write the timestamps of the I / O request entering the node and leaving the node to the memory address; The dwell time of the IO request on the corresponding node is calculated based on the timestamp of the same IO request in the same percpu variable.

[0006] Optionally, the method further includes: A lock-free circular buffer is created in the kernel for each block device. This lock-free circular buffer is a memory buffer shared by kernel mode and user mode. When an IO request leaves the corresponding IO stage, the IO leave event of that IO request is written to the lock-free circular buffer; The IO event data in the lockless circular buffer is output to the user space through the io_uring interface, which is an asynchronous IO interface provided by the Linux kernel.

[0007] Optionally, the method further includes: Register pre-map and post-map hooks in the core layer of the Linux kernel Device Mapper framework. The pre-map hook is a pre-callback function attached to the Device Mapper IO processing flow, and the post-map hook is a post-callback function attached to the Device Mapper IO processing flow. The execution timing of the pre-map hook is set to be before the Device Mapper core layer calls the target-driven map function; The execution timing of the post-map hook is set to be after the target driver's map function has been executed and before the dm_submit_bio function has been executed. The target driver is the driver module in the Device Mapper framework that implements the specific IO processing logic.

[0008] Optionally, setting the execution timing of the pre-map hook to be before the Device Mapper core layer calls the target-driven map function includes: When the Device Mapper core layer receives an IO request, the pre-map hook is triggered to execute; Create a DMIO tracing context for the parent bio corresponding to the IO request, and store the DMIO tracing context in the dm_io structure of the Device Mapper core layer. The DMIO tracing context is a structure that stores the parent IO tracing information, and the dm_io structure is a private IO request structure natively managed by the Device Mapper core layer. The atomic reference count of the DMIO tracing context is initialized to 1. The atomic reference count is a thread-safe reference count provided by the Linux kernel kref mechanism.

[0009] Optionally, the timing of setting the post-map hook to execute after the target-driven map function has finished executing and before the dm_submit_bio function has executed includes: After the target-driven map function completes execution, the post-map hook is triggered. Iterate through all the subbios generated by the map function, allocate a corresponding dm_io structure for each subbio, and inject the DMIO tracking context pointer corresponding to the parent bio into the dm_io structure of each subbio; After each pointer injection operation of a subbio is completed, the atomic reference count of the corresponding DMIO tracking context is incremented by 1.

[0010] Optionally, the step of collecting the dwell time of each IO request at the corresponding IO blocking node through the percpu variable at the preset IO blocking node of the Linux kernel storage stack includes: When a subbio enters a preset IO blocking node, the DMIO tracking context pointer is extracted from the dm_io structure corresponding to that subbio; Obtain the pointer to the parent IO trace structure in the DMIO trace context; Write the dwell time data of the subbio at the corresponding preset IO blocking node into the subIO statistics item of the parent IO tracking structure.

[0011] Optionally, after initializing the atomic reference count of the DMIO tracking context to 1, the process includes: When the subbio completes the IO processing, the subIO completion callback of the Device Mapper core layer is triggered, and the atomic reference count of the corresponding DMIO tracking context is decremented by 1; When the parent bio completes the IO processing, the parent IO completion callback of the Device Mapper core layer is triggered, and the atomic reference count of the corresponding DMIO tracking context is decremented by 1. When the atomic reference count of the DMIO tracing context reaches 0, the memory space corresponding to the DMIO tracing context is released.

[0012] Optionally, registering pre-map and post-map hooks in the core layer of the Linux kernel Device Mapper framework includes: When a subbio enters the IO processing flow of the next-layer Device Mapper device, the pre-map hook of the corresponding next-layer Device Mapper device is triggered to create a DMIO tracing context of the corresponding layer for the subbio. After the map function corresponding to the subbio is executed, the post-map hook of the next-level Device Mapper is triggered to inject the corresponding level of DMIO tracing context pointer into the next-level subbio generated by the subbio split from the subbio.

[0013] This application also discloses a Linux kernel I / O event handling anomaly detection system, including: The time consumption acquisition module is used to collect the residence time of each IO request at the corresponding IO blocking node by using the percpu variable at the preset IO blocking node of the Linux kernel storage stack. The preset IO blocking node is the execution node that causes IO request blocking during the bio phase and request phase of the Linux kernel storage stack. The data output module is used to create a preset file node in the kernel debugfs file system. When it receives a read request for the file node from the user space, it determines the total resident time in the percpu variable corresponding to all CPU cores and outputs the total resident time to the user space through the file node. The anomaly monitoring module is used to create a corresponding IO trace structure for each IO stage of each block device in the Linux kernel. When an IO request enters the corresponding IO stage, a structure instance based on the IO trace structure is generated and added to the corresponding trace list. When the IO request leaves the corresponding IO stage, the instance is removed from the list. The module monitors whether the IO request processing is abnormal in real time based on the total dwell time and the structure instance.

[0014] As can be seen from the above technical solution, this application pre-locks the key execution nodes that cause IO request blocking within the bio and request phases of the Linux kernel storage stack, and instrumentes the code at the entry and exit points of each blocking node. When the IO request reaches the instrumentation point, the acquisition logic obtains the current CPU core number and directly writes it to the memory variable independently maintained by that CPU core through the percpu variable mechanism, recording the timestamp of the IO request. Multi-core concurrent acquisition does not require locking. The kernel creates a file node in the debugfs file system and registers a read callback function. When the user space initiates a read operation, the callback function traverses the percpu variables of all online CPU cores, pairs and calculates the entry and exit timestamps of the same IO request distributed across different CPU cores, and outputs the total resident time covering the entire system. At the same time, an IO trace structure type is created for each block device's bio and request phases, and the trace list is maintained by a linked list: when an IO request enters a certain phase, the structure memory is allocated, the IO unique identifier and entry timestamp are filled in, and the structure is linked into the corresponding list; when the IO request leaves the phase, the structure is removed from the list and released. User-space tools read the total resident time data and combine it with the structure instances that have not been removed for a long time in the in-transit IO trace list to determine whether the IO request has experienced delay or blocking anomalies.

[0015] This application's time-consuming data collection operation only performs two timestamp recordings and one subtraction at the boundary of the blocked node. During normal operation, the processing logic of core I / O paths such as device drivers and schedulers remains unaffected. When user space actively reads the debugfs node, the callback function iterates through the percpu variables of all CPU cores to read the records as is. Based on the unique I / O identifier, it pairs the entry and exit timestamps of the same request scattered across different cores, completing the reconstruction of the total time consumption across CPUs and outputting the total resident time for anomaly detection. Simultaneously, the trace structure instance generated when an I / O request enters the bio stage is linked into the trace list for that stage, and the instance is removed upon normal exit. If an I / O request cannot obtain a hardware queue tag for an extended period, causing its structure to remain in the list, user space can discover the blocked instance by checking the list. Combining its entry timestamp and the peak total resident time of that stage, it directly determines that the I / O is in an abnormal blocked state and locates the corresponding execution node. By using percpu variables to record time without locking at critical blocking points in the storage stack, summarizing only during reads, and using structure instances to track the status of IO in transit, real-time detection and hierarchical delimitation of IO processing anomalies under normalized low-overhead deployment are achieved. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a Linux kernel I / O event handling anomaly detection method in an embodiment of this application; Figure 2 This is a schematic diagram of the process of collecting the dwell time of each IO request at the corresponding IO blocking node through the percpu variable at the preset IO blocking node of the Linux kernel storage stack in an embodiment of the present application. Figure 3 This is a flowchart illustrating the remaining steps of a Linux kernel I / O event handling anomaly detection method in an embodiment of this application; Figure 4 This is a flowchart illustrating the mounting steps of a Linux kernel I / O event handling anomaly detection method in an embodiment of this application. Figure 5 This is a flowchart illustrating the process of setting the execution timing of the pre-map hook in a Linux kernel IO event handling anomaly detection method according to an embodiment of this application, before the Device Mapper core layer calls the target driver's map function; Figure 6 This is a flowchart illustrating the process of setting the execution timing of the post-map hook in a Linux kernel IO event handling anomaly detection method according to an embodiment of this application, where the target driver's map function is executed after execution and the dm_submit_bio function is executed. Figure 7 This is a schematic diagram illustrating the process of registering pre-map hooks and post-map hooks in the core layer of the Linux kernel Device Mapper framework, as part of a Linux kernel IO event handling anomaly detection method in this application embodiment. Figure 8 This is a schematic diagram of the process of collecting the dwell time of each IO request at the corresponding IO blocking node through the percpu variable at the preset IO blocking node of the Linux kernel storage stack in an embodiment of the present application. Figure 9 This is a flowchart illustrating the process after the atomic reference count of the initialization DMIO tracking context is 1 in a Linux kernel IO event handling exception detection method according to an embodiment of this application. Figure 10 This is a schematic diagram of the structure of a Linux kernel I / O event handling anomaly detection system in an embodiment of this application. Detailed Implementation

[0018] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not limiting, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without such specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0019] In the embodiments of this application, specific details such as particular system structures and techniques are set forth in the following description for illustrative purposes only and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary details.

[0020] For example, consider a server running a database application. This server is equipped with a general-purpose processor, solid-state drives, and a Linux 5.15 kernel. It runs an online transaction processing (OLTP) database. During operation, periodic increases in response latency occur, requiring maintenance personnel to pinpoint the cause of the abnormal I / O performance. The maintenance personnel first use the system's built-in I / O statistics tool to collect system I / O performance data. This tool generates reports on disk read / write speeds, I / O request counts, and average wait times by reading statistics from the diskstats node in the proc file system. However, this report only shows the overall average disk I / O performance over a period of time and cannot distinguish whether the latency increase originates from the file system layer, the general block layer, or the device driver layer, nor can it capture millisecond-level instantaneous I / O latency fluctuations. The maintenance personnel then use a kernel tracing tool to enable I / O event tracing. Enabling this tool activates static tracing points in the block layer of the kernel, collecting all I / O events and writing them to log files. This collection process consumes significant CPU and memory resources, leading to increased latency in normal I / O processing for the database application, making it unsuitable for long-term use in a continuously running production environment. Operations and maintenance personnel are unable to pinpoint the specific kernel stack level where the IO latency anomaly occurs without affecting business operations, resulting in a long processing cycle for IO performance issues.

[0021] This application provides a method and system for detecting anomalies in Linux kernel I / O event processing. It deeply and structurally decomposes the Linux kernel storage stack I / O processing flow, dividing it into two core stages: the bio stage and the request stage. Furthermore, it identifies the key nodes causing I / O blocking in each stage. Kernel code-level instrumentation is then performed at the entry / exit points of each blocking point to accurately collect the time consumption of a single I / O request at each stage and each blocking node. A percpu variable is used as the storage medium for statistical data. Each CPU core maintains an independent counter and timestamp statistical variable. In multi-core concurrent scenarios, no locking is required, eliminating the performance overhead caused by lock contention. At the entry / exit points of each I / O blocking point, only the percpu variable of the local CPU is used for detection. Lock-free read and write operations are performed on rcpu variables, resulting in short instruction paths for data acquisition and low noise during normal operation, thus having no perceptible impact on business IO performance. Finally, file nodes are created through debugfs, and the kernel only aggregates and outputs the percpu variable data of all CPUs in real time when the user actively reads the node. There is no additional overhead for aggregation, reporting, or disk writing during normal operation. In addition, a dedicated trace data structure is designed for each IO stage of each device. The core fields include a unique identifier for the IO request (bio pointer address) and the timestamp of the IO entering the stage, providing a carrier for single IO lifecycle tracing. When IO enters a certain stage, a data structure instance is created and added to the trace list of the corresponding device and stage; when IO leaves the stage, the corresponding instance is removed from the list. This mechanism allows for real-time acquisition of: the dwell time of a single IO at each stage, the number of IOs currently in transit at each stage, and abnormal hungIOs that have been dwelling for a long time. It can identify abnormal IOs in a blocked state in real time without waiting for IO completion and directly delineate their kernel stack level, achieving second-level localization of sudden IO anomalies. Furthermore, a circular buffer is used as shared memory for events between kernel and user modes, constructing a lock-free message queue. When an IO leaves the stage, only the departure event is written to the queue. This lock-free operation avoids performance losses under high concurrency. User mode reads events from the circular buffer through the io_uring interface. Utilizing the asynchronous IO capabilities of io_uring, the context switching and memory copy overhead between user and kernel modes is significantly reduced. At this point, event writing and reading are completely asynchronous, and the execution path of business IO will not be blocked in high-concurrency scenarios, further reducing the performance impact of full event tracking. Thus, it can simultaneously meet the core requirements of low-overhead normalized operation and accurate layered observation of the entire IO chain.

[0022] This application discloses a method for detecting exceptions in Linux kernel I / O event handling, such as... Figure 1 As shown, it includes: S100: At the preset IO blocking node of the Linux kernel storage stack, the dwell time of each IO request at the corresponding IO blocking node is collected by the percpu variable. The preset IO blocking node is the execution node that causes IO request blocking during the bio phase and request phase of the Linux kernel storage stack. S200: Create a preset file node in the kernel debugfs file system. When a user space reads the file node, determine the total resident time in the percpu variable corresponding to all CPU cores, and output the total resident time to the user space through the file node. S300: Creates a corresponding IO trace structure for each IO stage of each block device in the Linux kernel. When an IO request enters the corresponding IO stage, it generates a structure instance based on the IO trace structure and adds it to the corresponding trace list. When the IO request leaves the corresponding IO stage, it removes the instance from the list. Based on the total dwell time and the structure instance, it monitors in real time whether the IO request processing is abnormal.

[0023] In this embodiment, the Linux kernel storage stack is a collection of software stacks within the Linux kernel responsible for handling I / O requests. From top to bottom, it includes the file system layer, the general block layer, and the device driver layer. The general block layer can be further divided into a bio stage and a request stage. The bio stage is the first stage of I / O request processing in the Linux kernel. During this stage, the kernel constructs a bio structure for each I / O request. The bio structure describes the relevant parameters of the I / O request, including the starting sector, read / write length, data buffer address, and completion callback function. The request stage is the second stage of I / O request processing in the Linux kernel. During this stage, the kernel converts the bio structure into a request structure. The request structure describes the I / O request queue items submitted to the block device, which are then sorted by the I / O scheduler and sent to the device driver layer for processing.

[0024] In this embodiment, the preset IO blocking nodes are execution nodes within the Linux kernel storage stack's bio and request phases that may cause IO requests to be blocked. These execution nodes are code execution locations in the IO request processing flow that may be paused due to factors such as insufficient resources, scheduling waits, and rate limiting policies. Within the bio phase, the preset IO blocking nodes may include submitbio execution nodes, blkthrottle rate-limiting execution nodes, rqqosthrottle scheduling execution nodes, and gettag tag acquisition execution nodes. Within the request phase, the preset IO blocking nodes may include request generation execution nodes, plug / unplug execution nodes, IO scheduler scheduling execution nodes, and hctx hardware queue acquisition execution nodes.

[0025] In this embodiment, the percpu variable is a memory variable independently maintained by each CPU core in the Linux kernel. Each CPU core can only perform write operations on its own corresponding percpu variable copy. The percpu variable copies between different CPU cores are independent of each other, and there is no scenario of shared writes. Therefore, in multi-core concurrent scenarios, write operations on percpu variables do not require locking, and there is no performance overhead caused by lock contention. The Linux kernel provides standard interfaces for the allocation, access, and release of percpu variables, including interfaces such as allocpercpu, freepercpu, and getcpuvar. In this embodiment, the management and access of percpu variables are implemented through the kernel standard interfaces.

[0026] In this embodiment, the debugfs file system is a virtual file system provided by the Linux kernel, used to export debugging information and runtime data of the kernel from kernel mode to user mode. The debugfs file system is usually mounted under the / sys / kernel / debug directory of the system. Kernel modules can create file directories and file nodes in the debugfs file system through the standard interface provided by debugfs, and set corresponding read and write callback functions for the file nodes. When a user-mode process performs read and write operations on a file node, the kernel will call the corresponding callback function to complete the data interaction.

[0027] In this embodiment, the IO tracking structure is a memory structure used to record the running status of a single IO request within a corresponding IO stage. The IO tracking structure contains a unique identifier for the IO request and a timestamp indicating when the IO request enters the corresponding IO stage. In one exemplary implementation, this structure can be defined as containing the following members: a void type `bio` pointer, using the address of the `bio` structure as the unique identifier for the IO request; a 64-bit unsigned integer `enter_time`, representing the high-precision timestamp of entering the stage; and a `structlist_head` type `list`, representing the linked list nodes added to the tracking list. The unique identifier for the IO request is identification information used to distinguish different IO requests. It can be the memory address of the `bio` structure or a globally unique sequence number generated by the kernel for each IO request. The timestamp indicating when the IO request enters the corresponding IO stage is the system time value obtained from the kernel time source when the IO request enters the IO stage. The kernel time source can be the kernel's global jiffies count or a nanosecond-level time value corresponding to a high-precision timer.

[0028] In this embodiment, the dwell time of each I / O request at the corresponding I / O blocking node is collected using the percpu variable at the preset I / O blocking node in the Linux kernel storage stack. Specifically, during the kernel compilation stage, instrumentation is performed on the kernel code execution location corresponding to the preset I / O blocking node. Instrumentation refers to inserting preset collection and processing functions at the code entry and exit points of the preset I / O blocking node. When the I / O request processing flow reaches the entry point of the preset I / O blocking node, the collection and processing function at the entry point is triggered; when the I / O request processing flow reaches the exit point of the preset I / O blocking node, the collection and processing function at the exit point is triggered.

[0029] In this embodiment, a set of independent percpu variables is allocated for each preset IO blocking node. These percpu variables store the entry and exit timestamps of each IO request at the corresponding preset IO blocking node. The allocation of percpu variables is completed during the kernel module initialization phase. During kernel module initialization, the kernel-provided allocpercpu interface allocates percpu variables of the corresponding type to each preset IO blocking node. The percpu variables can be of 64-bit unsigned integer type, used to store nanosecond-level timestamp values. During kernel module unloading, the kernel-provided freepercpu interface releases all allocated percpu variables to avoid memory leaks.

[0030] In alternative implementations, such as Figure 2As shown, the step of collecting the dwell time of each IO request at the corresponding IO blocking node through the percpu variable at the preset IO blocking node in the Linux kernel storage stack includes: S110: Determine the CPU core number currently executing the IO request and obtain the memory address of the percpu variable corresponding to that CPU core; S120: Write the timestamp of the IO request entering the node and the timestamp of the IO request leaving the node to the memory address; S130: Calculate the dwell time of the IO request on the corresponding node based on the timestamp of the same IO request in the same percpu variable.

[0031] In this embodiment, when the IO request processing flow reaches the entry point of the preset IO blocking node, the acquisition and processing function at the entry point first obtains the bio structure or request structure corresponding to the current IO request, extracts the unique identifier of the IO request, then obtains the CPU core number where the current execution context resides, obtains the memory address of the percpu variable corresponding to the CPU core through the getcpuvar interface, and then obtains the current system timestamp from the kernel's high-precision time source. This timestamp, along with the unique identifier of the IO request, is written into the memory address of the corresponding percpu variable. After the writing operation is completed, the acquisition and processing function finishes execution, and the kernel continues to execute the original processing logic of the preset IO blocking node.

[0032] In this embodiment, when the IO request processing flow completes the original processing logic of the preset IO blocking node and reaches the node's exit position, the acquisition processing function at the exit first obtains the unique identifier of the current IO request, then obtains the CPU core number where the current execution context resides, obtains the memory address of the percpu variable corresponding to that CPU core through the getcpuvar interface, and then obtains the current system timestamp from the kernel's high-precision time source. This timestamp, along with the unique identifier of the IO request, is written into the memory address of the corresponding percpu variable. After the write operation is completed, the acquisition processing function calculates the dwell time of the IO request at the current preset IO blocking node based on the entry and exit timestamps corresponding to the same unique identifier of the IO request. The dwell time is calculated by subtracting the entry timestamp from the exit timestamp value. The calculated dwell time is then written into the memory address of the corresponding percpu variable. After this, the acquisition processing function completes its execution, and the kernel continues to execute subsequent IO processing logic.

[0033] In this embodiment, a preset file node is created in the kernel debugfs file system. When a read request for this file node is received from user space, the total resident time in the percpu variable corresponding to all CPU cores is determined, and the total resident time is output to user space through this file node. Specifically, the creation of the preset file node is completed during the kernel module initialization phase. During kernel module initialization, a corresponding directory is first created in the root directory of the debugfs file system. The directory name can be set according to the actual implementation scenario. Then, a preset file node is created in this directory. The file node type is a regular file, and a corresponding file operation structure is set for this file node. The file operation structure at least contains a read callback function, which is used to handle read requests for this file node from user-space processes.

[0034] In this embodiment, when a user-mode process performs a read operation on a preset file node via the `read` system call, the kernel triggers the execution of the `read` callback function corresponding to the preset file node. During the execution of the `read` callback function, it first traverses all online CPU cores in the system. For each CPU core, it obtains the memory address of the percpu variable corresponding to that CPU core through the `getcpuvaroffline` interface, reads the dwell time data of all IO requests stored in that memory address, and stores the read data in a temporary buffer in kernel mode. After completing the reading of the percpu variable data of all CPU cores, the `read` callback function performs summary processing on all data in the temporary buffer. The summary processing first traverses the percpu variable records of all CPUs, performs hash classification based on the unique identifier of the IO request carried by each record, pairs the entry and exit timestamps of the same IO request that are physically scattered in different CPU core variables, calculates the dwell time of the IO request on each blocked node, merges them by node, and finally sorts and organizes them according to a preset format. This step ensures that even if the processing of an I / O request is scheduled to be executed on multiple CPUs, its complete blocking time can still be accurately reconstructed. Data is sorted according to the unique identifier of the I / O request and formatted according to a preset format. After the summary processing is complete, the read callback function copies the summarized data in the temporary buffer to the user buffer provided by the user-space process through the kernel-provided copy_to_user interface, while setting the number of bytes returned in this read operation, thus completing the user-space read request processing.

[0035] In this embodiment, a corresponding IO trace structure is created for each IO stage of each block device in the Linux kernel. When an IO request enters the corresponding IO stage, a structure instance based on the IO trace structure is generated and added to the corresponding trace list. When the IO request leaves the corresponding IO stage, the instance is removed from the list. The IO request processing is monitored in real time for anomalies based on the total dwell time and the number of structure instances. Specifically, each block device in the system corresponds to an independent device trace structure. The device trace structure contains the device number, device name, and the head nodes of the trace lists for the two IO stages: the bio stage and the request stage. Each trace list head node corresponds to a doubly linked circular list, and each node in the list corresponds to an IO trace structure instance. The creation of the device trace structure is completed during block device initialization. When the system detects a new block device registration, it creates a corresponding device trace structure for that block device and initializes the trace lists for the two IO stages. When a block device is deregistered from the system, the device trace structure corresponding to that block device, as well as all remaining IO trace structure instances in the trace list, are released to prevent memory leaks.

[0036] In this embodiment, when an IO request enters the corresponding IO stage, the kernel allocates memory space for the IO request, generates an instance of an IO trace structure, initializes the fields in the instance, writes the unique identifier of the IO request and the timestamp of entering the corresponding IO stage into the corresponding fields of the instance, and then adds the IO trace structure instance to the trace list of the corresponding IO stage of the corresponding block device. When the IO request completes all processing logic of the corresponding IO stage and leaves the IO stage, the kernel searches for the IO trace structure instance corresponding to the IO request in the corresponding trace list, removes the instance from the trace list, and releases the memory space occupied by the instance.

[0037] In this embodiment, the process of real-time monitoring of whether IO request processing is abnormal is performed by a user-space process. The user-space process periodically reads the total resident time data from the preset file nodes in debugfs, and simultaneously reads the in-transit IO trace list data exported by the kernel. The user-space process first analyzes the total resident time data, identifies IO requests whose resident time exceeds a preset threshold, and marks them as potentially abnormal IO. Subsequently, the user-space process traverses the in-transit IO trace list, searching for IO trace structure instances in the list whose time exceeds the preset threshold. The IO requests corresponding to these instances have not left the corresponding IO stage for a long time and are marked as blocked abnormal IO. The user-space process combines the total resident time data and the in-transit IO trace list data to determine the IO stage and blocking node where the abnormal IO request is located, thus completing the detection and location of IO anomalies.

[0038] In this embodiment, the CPU core number currently executing the I / O request is determined, and the memory address of the percpu variable corresponding to that CPU core is obtained. Specifically, determining the CPU core number currently executing the I / O request can be achieved using the kernel-provided `getcpu` function. The `getcpu` function returns the CPU core number of the current execution context and the NUMA node number corresponding to the current CPU core. Calling the `getcpu` function within the acquisition and processing function allows the acquisition of the currently executing CPU core number. It should be noted that the execution of the `getcpu` function does not trigger process scheduling or cause CPU core switching. Therefore, the CPU core number obtained through the `getcpu` function is completely consistent with the CPU core number where the current acquisition and processing function is executing, ensuring the accuracy of percpu variable access.

[0039] In this embodiment, obtaining the memory address of the percpu variable corresponding to the CPU core can be achieved through the getcpuvar interface provided by the kernel. The getcpuvar interface returns the memory address of the percpu variable copy corresponding to the current CPU core based on the passed percpu variable base address and the current CPU core number. For each preset IO blocking node, a percpu variable base address is pre-allocated. In the acquisition and processing function, the percpu variable base address of the corresponding preset IO blocking node is passed to the getcpuvar interface to obtain the memory address of the percpu variable copy corresponding to the current CPU core.

[0040] In this embodiment, the timestamps of the IO request entering and leaving the node are written to the memory address. Specifically, in the entry acquisition and processing function of the preset IO blocking node, the unique identifier of the IO request is written to the first field of the percpu variable memory address, and the timestamp of entering the node is written to the second field. In the exit acquisition and processing function of the preset IO blocking node, the corresponding IO request is first matched according to the unique identifier of the IO request stored in the percpu variable memory address, and then the timestamp of leaving the node is written to the third field. The timestamp is obtained through the ktimeget function provided by the kernel. The ktimeget function returns a nanosecond-level timestamp corresponding to the kernel's high-precision timer. The accuracy of this timestamp can reach the nanosecond level, which can meet the acquisition accuracy requirements for IO request dwell time.

[0041] In this embodiment, the dwell time of the IO request on the corresponding node is calculated based on the timestamp of the same IO request in the same percpu variable. Specifically, the egress acquisition and processing function reads the entry and exit timestamps stored in the memory address of the percpu variable, subtracts the entry timestamp from the exit timestamp value, and the difference is the dwell time of the IO request on the current preset IO blocking node. The calculated dwell time is written to the fourth field of the percpu variable's memory address, and simultaneously written to the IO tracking structure instance corresponding to the IO request, achieving unified management of the end-to-end dwell time data for a single IO request.

[0042] In alternative implementations, such as Figure 3 As shown, the method further includes: S400: Create a lock-free circular buffer for each block device in the kernel. The lock-free circular buffer is a memory buffer shared by kernel mode and user mode. S500: When an IO request leaves the corresponding IO stage, write the IO leave event of the IO request to the lock-free circular buffer; S600: Outputs the IO event data in the lockless circular buffer to the user space through the io_uring interface, which is an asynchronous IO interface provided by the Linux kernel.

[0043] In this embodiment, a lock-free circular buffer is created in the kernel for each block device. This lock-free circular buffer is a shared memory buffer between kernel mode and user mode. The lock-free circular buffer is a memory buffer implemented using a circular queue structure. The memory space of the buffer is allocated by the kernel mode, and through memory mapping, the buffer's memory space is mapped to the virtual address space of the user-mode process. This allows kernel mode and user mode to directly access the same physical memory space without memory copying operations. The lock-free circular buffer adopts a producer-consumer model. The kernel mode acts as the producer, writing IO event data into the buffer, while the user mode acts as the consumer, reading IO event data from the buffer. Through memory barriers and sequence number counting mechanisms, concurrent read and write operations are achieved in a lock-free environment, avoiding the lock contention overhead in multi-core concurrent scenarios.

[0044] In this embodiment, a lock-free circular buffer is created in the kernel for each block device. Specifically, after the block device initialization is complete, the kernel determines the size of the lock-free circular buffer based on the block device's queue depth and the expected IOPS value. The buffer size is typically set to a power of 2 to simplify the modulo operation of the circular queue. After determining the buffer size, the kernel allocates contiguous physical memory pages through the `allocpages` interface, mapping these physical memory pages to the kernel-mode virtual address space, thus completing the kernel-mode memory initialization of the lock-free circular buffer. Simultaneously, the kernel sets a corresponding memory mapping function for this lock-free circular buffer. When a user-mode process performs a memory mapping operation on the file node corresponding to the buffer via the `mmap` system call, the kernel maps the physical memory pages of the buffer to the virtual address space of the user-mode process, completing the user-mode memory mapping.

[0045] In this embodiment, when an IO request leaves its corresponding IO stage, an IO leave event for that IO request is written to the lock-free circular buffer. Specifically, when an IO request completes all processing logic for its corresponding IO stage and leaves that stage, the kernel generates a corresponding IO leave event. This event includes a unique identifier for the IO request, a timestamp indicating the IO request left the corresponding IO stage, the dwell time of all preset IO blocking nodes within the IO stage, the block device number corresponding to the IO request, and the read / write type and data length of the IO request. After generating the IO leave event, the kernel first obtains the current producer sequence number of the lock-free circular buffer. Based on the producer sequence number and the buffer size, it calculates the memory address to which the event is written, writes the IO leave event data to the corresponding memory address, and then uses a write memory barrier instruction to ensure that the event data is visible to other CPU cores. Finally, it increments the producer sequence number atomically to complete the write. When a user-mode consumer reads, it first reads the consumer sequence number, using a read memory barrier to ensure that the sequence number is read before the data is read, and then safely retrieves the event data from the buffer memory address based on the sequence number. This lock-free sequence control ensures data consistency under multi-core concurrency.

[0046] In this embodiment, I / O event data in a lock-free circular buffer is output to user space via the io_uring interface, which is an asynchronous I / O interface provided by the Linux kernel. User-space processes submit a batch of asynchronous read requests to the kernel in advance through the io_uring interface. These requests point to a preset file node associated with the lock-free circular buffer. When new I / O exit events become available for reading in the circular buffer, the kernel processes these pre-submitted read requests, directly copies the event data to the buffer specified by the user space, and returns the number of bytes successfully read through the io_uring completion queue. This method, where user space actively pulls data, conforms to the asynchronous I / O model of io_uring and reduces the context switching overhead required for the kernel to actively push data to user space.

[0047] In alternative implementations, such as Figure 4 As shown, the method further includes: S810: Register pre-map hooks and post-map hooks in the core layer of the Linux kernel Device Mapper framework. The pre-map hook is a pre-callback function attached to the Device Mapper IO processing flow, and the post-map hook is a post-callback function attached to the Device Mapper IO processing flow. S820: Set the execution timing of the pre-map hook to before the Device Mapper core layer calls the target-driven map function; S830: The execution timing of the post-map hook is set to after the target driver's map function is executed and before the dm_submit_bio function is executed. The target driver is the driver module in the Device Mapper framework that implements the specific IO processing logic.

[0048] In this embodiment, pre-map and post-map hooks are registered in the core layer of the Linux kernel Device Mapper framework. Pre-map hooks are pre-callback functions attached to the Device Mapper IO processing flow, and post-map hooks are post-callback functions attached to the Device Mapper IO processing flow. The Device Mapper framework is a block device virtualization framework provided by the Linux kernel. Through this framework, virtual block devices can be created in the kernel. The IO processing logic of the virtual block devices is implemented by the corresponding target driver. The Device Mapper framework adopts a layered architecture of a core layer plus target drivers. The core layer is responsible for the management of virtual block devices and the reception and distribution of IO requests. The target drivers are responsible for implementing the specific IO processing logic, including cloning, splitting, mapping, verification, and encryption of IO requests. Common target drivers include multipath drivers, crypt drivers, verity drivers, thin drivers, and RAID drivers. Different target drivers can implement different block device virtualization functions.

[0049] In this embodiment, the core layer IO processing entry point of the Device Mapper framework is the `dmmakerequest` function. When an upper-layer application submits an IO request to a virtual block device created by the Device Mapper, the block layer routes the IO request to the `dmmakerequest` function. The `dmmakerequest` function performs IO request validity verification, obtains the target driver instance corresponding to the virtual block device, locks the target driver instance, and then calls the `map` function of the target driver to process the IO request. The `map` function of the target driver clones or splits the bio structure corresponding to the original IO request according to its own business logic, generating one or more sub-bio structures. The sub-bio structures correspond to the underlying physical block devices, and finally, the `dmsubmitbio` function submits the sub-bio structures to the general block layer processing of the underlying physical block devices.

[0050] In this embodiment, pre-map and post-map hooks are registered in the core layer of the Linux kernel Device Mapper framework. Specifically, hook registration is implemented using the Linux kernel's native notifier notification chain mechanism. The notifier notification chain mechanism is an event subscription and notification mechanism provided by the Linux kernel. Kernel modules can subscribe to specific events in the kernel by registering callback functions for the notification chain. When an event occurs, the kernel iterates through all registered callback functions in the notification chain and triggers their execution. In this embodiment, pre-map and post-map event notification chains are created in the core layer of the Device Mapper framework. The callback functions corresponding to the pre-map hooks are registered in the pre-map event notification chain, and the callback functions corresponding to the post-map hooks are registered in the post-map event notification chain. After hook registration is complete, when the IO processing flow of the Device Mapper core layer reaches the corresponding position, the event notification of the corresponding notification chain is triggered, and all registered callback functions in the notification chain are iterated to complete the hook execution.

[0051] It should be noted that the above embodiments can be based on open-source Linux kernel source code, with pre-built notification chain call points added to critical paths such as the dm_make_request function in the DM core layer. This small amount of low-intrusion patching of the kernel mainline is a common practice in kernel module development in this field. The kernel observation module of this application can directly call custom registration functions such as register_dm_pre_map_notifier to mount callbacks and achieve hook injection without the need for heavy binary rewriting of the running kernel.

[0052] In this embodiment, the execution timing of the pre-map hook is set before the Device Mapper core layer calls the target driver's map function. Specifically, in the dmmakerequest function of the Device Mapper core layer, after completing IO request validity verification, obtaining the target driver instance, and locking the target driver instance, a trigger code for the pre-map event notification chain is added at the code location before the target driver's map function is called. When the code executes to this location, the pre-map event notification chain is triggered, calling all registered pre-map hook callback functions to complete the execution of the pre-map hook.

[0053] In this embodiment, the execution timing of the post-map hook is set to after the target driver's map function has finished executing and before the dm_submit_bio function has executed. The target driver is the driver module in the Device Mapper framework that implements the specific IO processing logic. Specifically, in the dmmakerequest function of the Device Mapper core layer, after the target driver's map function has finished executing and returned, before calling the dmsubmitbio function to submit the sub-bio structure, trigger code for the post-map event notification chain is added at this code location. When the code executes to this location, the post-map event notification chain is triggered, calling all registered post-map hook callback functions to complete the execution of the post-map hook.

[0054] In this embodiment, when the Device Mapper core layer receives an IO request, the pre-map hook is triggered. Specifically, the upper-layer service submits an IO request to the Device Mapper virtual block device. The block layer routes the IO request to the dmmakerequest function in the Device Mapper core layer. After the dmmakerequest function completes the IO request validity verification, obtains the target driver instance, and locks the target driver instance, it triggers the pre-map event notification chain and executes the pre-map hook callback function registered in this application. The input parameters of the pre-map hook callback function include a pointer to the bio structure corresponding to the current IO request and a pointer to the corresponding dm_io structure.

[0055] In alternative implementations, such as Figure 5 As shown, setting the execution timing of the pre-map hook to be before the Device Mapper core layer calls the target-driven map function includes: S010: When the Device Mapper core layer receives an IO request, the pre-map hook is triggered to execute; S020: Create a DMIO tracing context for the parent bio corresponding to the IO request, and store the DMIO tracing context in the dm_io structure of the Device Mapper core layer. The DMIO tracing context is a structure that stores the parent IO tracing information, and the dm_io structure is a private IO request structure natively managed by the Device Mapper core layer. S030: Initialize the atomic reference count of the DMIO tracing context to 1. The atomic reference count is a thread-safe reference count provided by the Linux kernel kref mechanism.

[0056] In this embodiment, a DMIO tracing context is created for the parent bio corresponding to the IO request. The DMIO tracing context is stored in the `dm_io` structure of the Device Mapper core layer. The DMIO tracing context is a structure that stores parent IO tracing information, and the `dm_io` structure is a private IO request structure natively managed by the Device Mapper core layer. The DMIO tracing context is a structure that stores parent IO tracing information, and it may contain, but is not limited to, the following fields: a pointer to the parent IO tracing structure `parent_track`, a globally unique identifier for the parent IO `parent_id`, the DM device's major and minor device numbers, the target driver type, an atomic reference count `refcount` using the kernel's kref mechanism, and the child bio tracing entry list head node `child_list`.

[0057] In this embodiment, the `dm_io` structure is a natively defined structure in the Device Mapper core layer, used to manage the private data of each IO request in the Device Mapper framework. This includes the virtual block device information corresponding to the IO request, the target driver instance information, the IO request completion callback function, and the private data fields used by the target driver. Each IO request entering the Device Mapper core layer corresponds to an independent `dm_io` structure instance. The lifecycle of the `dm_io` structure instance is completely consistent with the lifecycle of the corresponding IO request. When the IO request completes all processing logic, the corresponding `dm_io` structure instance is released.

[0058] In this embodiment, the DMIO tracing context is a memory structure used to store parent IO tracing information in the Device Mapper framework. The DMIO tracing context structure includes a parent IO tracing structure pointer, a parent IO globally unique identifier, the DM device major device number, the DM device minor device number, the target driver type, an atomic reference count, and the head node of the child bio tracing entry list. The parent IO tracing structure pointer points to the IO tracing structure instance corresponding to the parent IO request. The parent IO globally unique identifier is consistent with the unique identifier of the parent IO request. The DM device major and minor device numbers correspond to the device numbers of the Virtual Block Device of the Device Mapper to which the IO request belongs. The target driver type corresponds to the type identifier of the target driver handling the IO request. The atomic reference count is used to manage the lifecycle of the DMIO tracing context, and the head node of the child bio tracing entry list is used to manage all tracing entries corresponding to the child bios generated by splitting the parent IO request.

[0059] In this embodiment, a DMIO tracing context is created for the parent bio corresponding to the IO request. In the specific implementation process, when the pre-map hook callback function is executed, it first obtains the parent bio structure pointer from the input parameters, extracts the unique identifier of the parent IO request, then allocates memory space through the kernel's slab cache pool, creates an instance of the DMIO tracing context, initializes each field in the instance, writes the unique identifier of the parent IO request into the parent IO global unique identifier field, writes the major device number and minor device number of the current Device Mapper virtual block device into the corresponding fields, writes the type identifier of the target driver that handles the IO request into the target driver type field, finds the corresponding parent IO tracing structure instance through the unique identifier of the parent IO request, writes the pointer of the instance into the parent IO tracing structure pointer field, and initializes the head node of the child bio tracing entry list.

[0060] In this embodiment, the DMIO tracing context is stored in the dm_io structure of the Device Mapper core layer. In specific implementation, the dm_io structure contains private data extension fields used by the target driver. The pre-map hook callback function writes the pointer of the created DMIO tracing context instance into the private data extension field of the dm_io structure, so that the DMIO tracing context instance is bound to the dm_io structure instance corresponding to the current IO request. In the subsequent IO processing flow, the pointer of the DMIO tracing context instance can be extracted from the dm_io structure at any time.

[0061] In this embodiment, the atomic reference count of the DMIO tracking context is initialized to 1. This atomic reference count is a thread-safe reference count provided by the Linux kernel's kref mechanism. The Linux kernel's kref mechanism is a thread-safe atomic reference counting mechanism provided by the kernel. It increments and decrements the reference count through atomic operations, ensuring the accuracy of the reference count in multi-core concurrent scenarios. When the reference count decreases to 0, a preset release function is automatically called to release the corresponding memory object. The kref mechanism is widely used in the lifecycle management of various memory objects in the Linux kernel to ensure that memory objects are not prematurely released during use, avoiding the useafterfree memory safety problem.

[0062] In this embodiment of the application, the atomic reference count of the DMIO tracking context is initialized to 1. In the specific implementation process, the atomic reference count in the DMIO tracking context instance is initialized through the krefinit function provided by the kernel kref mechanism. The initial value of the reference count is set to 1. The initial value of 1 corresponds to the lifecycle of the parent bio itself. When the parent bio finishes processing, it will perform a decrement operation on the reference count.

[0063] In this embodiment, after the `map` function of the `target` driver completes execution, the `post-map` hook is triggered. Specifically, the Device Mapper core layer calls the `map` function of the `target` driver to process the IO request. After the `map` function completes execution and returns, the Device Mapper core layer triggers the `post-map` event notification chain, executing the `post-map` hook callback function registered in this application. The input parameters of the `post-map` hook callback function include a pointer to the parent bio structure corresponding to the current IO request, a pointer to the `dm_io` structure, and a list of all child bio structures generated by the `map` function.

[0064] In this embodiment, all subbios generated by the map function are traversed, and a corresponding dm_io structure is allocated to each subbio. The DMIO tracking context pointer corresponding to the parent bio is injected into the dm_io structure of each subbio. Specifically, the post-map hook callback function iterates through the list of subbio structures passed in as input. For each subbio structure in the list, the Device Mapper core layer allocates an independent dm_io structure instance to manage the subbio's private data within the Device Mapper framework. The post-map hook callback function obtains a pointer to the dm_io structure instance corresponding to each subbio, preparing for subsequent pointer injection operations.

[0065] In this embodiment, the DMIO tracking context pointer corresponding to the parent bio is injected into the dm_io structure of each child bio. Specifically, the post-map hook callback function extracts the pointer to the DMIO tracking context instance from the dm_io structure corresponding to the parent bio and writes this pointer into the private data extension field of the dm_io structure corresponding to each child bio. This ensures that the dm_io structures corresponding to all child bios are bound to the DMIO tracking context instance created by the parent bio. In the subsequent processing flow of the child bio, the pointer to the DMIO tracking context instance corresponding to the parent bio can be extracted from its own dm_io structure at any time.

[0066] In alternative implementations, such as Figure 6 As shown, the timing for setting the post-map hook is after the target driver's map function has finished executing and before the dm_submit_bio function has executed, including: S040: After the target-driven map function completes execution, the post-map hook is triggered. S050: Traverse all subbios generated by the map function, allocate a corresponding dm_io structure for each subbio, and inject the DMIO tracking context pointer corresponding to the parent bio into the dm_io structure of each subbio; S060: After each pointer injection operation of a sub-bio is completed, the atomic reference count of the corresponding DMIO tracking context is incremented by 1.

[0067] In this embodiment, after each subbio pointer injection operation is completed, the atomic reference count of the corresponding DMIO tracking context is incremented by 1. Specifically, after each subbio DMIO tracking context pointer injection operation is completed, the krefget function provided by the kernel kref mechanism performs an atomic increment operation on the atomic reference count of the DMIO tracking context instance. The incremented reference count is consistent with the number of subbios to which the pointer is injected. Each subbio's reference count is incremented by 1, ensuring that the lifecycle of each subbio corresponds to the reference count of the DMIO tracking context instance, thus preventing the DMIO tracking context instance from being prematurely released before the subbio processing is complete.

[0068] In this embodiment, when a subbio enters a preset IO blocking node, the DMIO tracking context pointer is extracted from the dm_io structure corresponding to the subbio. Specifically, the subbio structure is submitted to the underlying physical block device's general block layer processing via the dmsubmitbio function. When the subbio's processing reaches the preset IO blocking node, the acquisition and processing function at the node is triggered. This function first obtains the dm_io structure pointer corresponding to the subbio structure and then extracts the pointer to the DMIO tracking context instance corresponding to the parent bio from the private data extension field of the dm_io structure.

[0069] In this embodiment, the pointer to the parent IO trace structure in the DMIO trace context is obtained. Specifically, the acquisition and processing function reads the parent IO trace structure pointer field from the extracted DMIO trace context instance to obtain the memory address of the IO trace structure instance corresponding to the parent IO request for that child bio. This IO trace structure instance stores the full-link tracing information of the parent IO request, as well as the storage area for the child IO statistics.

[0070] In this embodiment, the dwell time data of the sub-bio at the corresponding preset IO blocking node is written into the sub-IO statistics item of the parent IO tracking structure. Specifically, the acquisition and processing function first calculates the dwell time data of the sub-bio at the current preset IO blocking node, and simultaneously obtains the device number of the underlying physical block device corresponding to the sub-bio, the read / write type of the sub-bio, and the unique identifier of the sub-bio. This information, along with the dwell time data, forms a sub-IO statistics entry, which is then written into the sub-IO statistics chain table of the parent IO tracking structure, completing the synchronization of the sub-bio dwell time data with the parent IO tracking structure. Through this step, the dwell time data of all sub-bios at each preset IO blocking node will be synchronized to the corresponding parent IO tracking structure, achieving unified management of the end-to-end dwell time data of all sub-IO requests generated by splitting the parent IO request.

[0071] In this embodiment, when the sub-bio completes IO processing, a sub-IO completion callback of the Device Mapper core layer is triggered, decrementing the atomic reference count of the corresponding DMIO tracking context by 1. Specifically, after the sub-bio structure completes all IO processing logic in the underlying physical block device, it triggers the corresponding biendio completion callback function. This callback function is provided by the Device Mapper core layer. When the callback function executes, it first completes the finalization of the dm_io structure corresponding to the sub-bio, and then uses the krefput function provided by the kernel kref mechanism to perform an atomic decrement operation on the atomic reference count in the DMIO tracking context instance, thus completing the reference count update corresponding to the sub-bio's lifecycle.

[0072] In this embodiment, when the parent bio completes IO processing, the parent IO completion callback of the Device Mapper core layer is triggered, decrementing the atomic reference count of the corresponding DMIO tracking context by 1. Specifically, after all child bios corresponding to the parent bio have completed IO processing, the Device Mapper core layer triggers the biendio completion callback function corresponding to the parent bio. When the callback function executes, it first completes the cleanup of the dm_io structure corresponding to the parent bio, and then uses the krefput function provided by the kernel kref mechanism to perform an atomic decrement operation on the atomic reference count in the DMIO tracking context instance, completing the reference count update corresponding to the parent bio's lifecycle.

[0073] In this embodiment, the memory space corresponding to the DMIO tracking context is released when the atomic reference count of the DMIO tracking context reaches 0. Specifically, during the initialization of the atomic reference count of the DMIO tracking context instance, a release function corresponding to the reference count reaching 0 is simultaneously set. When the krefput function executes and detects that the atomic reference count has decreased to 0, the preset release function is automatically called. The release function first traverses the child bio tracking entry list in the DMIO tracking context instance, releasing the memory space occupied by all child IO statistics entries in the list. Then, it releases the memory space occupied by the DMIO tracking context instance itself, returning the memory space to the corresponding slab cache pool, thus completing the memory release operation for the entire DMIO tracking context instance.

[0074] In alternative implementations, such as Figure 7 As shown, registering pre-map and post-map hooks in the core layer of the Linux kernel Device Mapper framework includes: S811: When a subbio enters the IO processing flow of the next-level Device Mapper device, the pre-map hook of the corresponding next-level Device Mapper device is triggered to create a DMIO tracing context of the corresponding level for the subbio. S812: After the map function corresponding to the subbio is executed, the post-map hook of the device corresponding to the next-level Device Mapper is triggered to inject the corresponding level of DMIO tracing context pointer into the next-level subbio generated by the subbio split.

[0075] In this embodiment, when a sub-bio enters the IO processing flow of the next-layer Device Mapper device, the pre-map hook of the corresponding next-layer Device Mapper device is triggered to create a DMIO tracing context for the sub-bio at the corresponding level. Multi-layer Device Mapper device stacking refers to an architecture where multiple Device Mapper virtual block devices are nested layer by layer. The underlying physical device of the upper-layer Device Mapper virtual block device is another virtual block device created by the lower-layer Device Mapper framework. IO requests need to be processed through multiple layers of Device Mapper virtual block devices before finally being submitted to the physical storage device. Common multi-layer stacking architectures include a two-layer architecture with dmcrypt driver superimposed on dmmultipath driver, a two-layer architecture with dmthin driver superimposed on dmraid driver, and more nested architectures with multiple layers.

[0076] In this embodiment, when a sub-bio enters the IO processing flow of the next-layer Device Mapper device, the pre-map hook of the corresponding next-layer Device Mapper device is triggered to create a DMIO tracing context for the sub-bio at the corresponding level. Specifically, the sub-bio generated by the upper-layer Device Mapper device corresponds to a virtual block device created by the next-layer Device Mapper. After the sub-bio is submitted via the `dmsubmitbio` function, it enters the IO processing flow of the next-layer Device Mapper device, triggering the execution of the pre-map hook corresponding to that device. During the execution of the pre-map hook, the sub-bio is used as the parent bio of the current level, and a DMIO tracing context instance corresponding to the current level is created for it. This instance records the Device Mapper device information and target driver type of the current level, as well as a pointer to the DMIO tracing context instance of the upper-layer device, establishing the association between multiple levels of DMIO tracing contexts.

[0077] In this embodiment, after the map function corresponding to the sub-bio is executed, the post-map hook of the corresponding next-layer Device Mapper device is triggered, injecting the corresponding level's DMIO tracking context pointer into the next-level sub-bio generated by the sub-bio split from the sub-bio. Specifically, after the target driver map function of the next-layer Device Mapper device is executed, the corresponding post-map hook is triggered. The post-map hook iterates through the next-level sub-bios generated by the map function, allocates a corresponding dm_io structure for each next-level sub-bio, injects the pointer of the current level's DMIO tracking context instance into the dm_io structure of each next-level sub-bio, and simultaneously increments the atomic reference count of the current level's DMIO tracking context instance. Through this step, in a multi-layer Device Mapper stacking architecture, each level's IO request can establish a corresponding tracking context, achieving full-link tracking of the multi-layer IO processing flow. Simultaneously, through pointer association between levels, synchronization of sub-IO time consumption data at all levels with the tracking structure of the top-level original business IO request is achieved.

[0078] In alternative implementations, such as Figure 8 As shown, the step of collecting the dwell time of each IO request at the corresponding IO blocking node through the percpu variable at the preset IO blocking node in the Linux kernel storage stack includes: S140: When a subbio enters a preset IO blocking node, the DMIO tracking context pointer is extracted from the dm_io structure corresponding to the subbio; S150: Obtain the pointer to the parent IO trace structure in the DMIO trace context; S160: Write the dwell time data of the subbio at the corresponding preset IO blocking node into the subIO statistics item of the parent IO tracking structure.

[0079] For example, consider a server running database services. This server is equipped with a general-purpose architecture processor, a dual-port host bus adapter card, and a dual-active storage array connected via a fiber optic switch. The storage array maps to two logical unit numbers, corresponding to two physical paths on the server, with device names sda and sdb. The server operating system uses a Linux 5.15 kernel, and is configured with a multi-path aggregation device named mpatha. It employs a queue length path load balancing strategy, with Kyber as the IO scheduler. The mpatha device is formatted as an ext4 file system and mounted to the data directory, where online transaction processing (OLTP) database services are deployed. The server has deployed the IO event handling anomaly detection method disclosed in this application, completed the instrumentation of preset IO blocking nodes in the general block layer bio and request phases, completed the allocation and initialization of percpu variables, completed the creation of debugfs preset file nodes, completed the initialization of the IO tracing structure management mechanism, and completed the registration of the pre-map hooks and post-map hooks in the core layer of the Device Mapper framework.

[0080] The database service initiates a 32KB synchronous write I / O to the database file in the data directory via the pwrite system call. After processing by the virtual file system layer and the ext4 file system layer, a corresponding raw bio structure is generated. The kernel then calls the submitbio function to submit this raw bio structure to the block layer. The block device corresponding to this raw bio structure is an mpatha virtual block device created through multipathing. Therefore, the block layer routes this I / O request to the dmmakerequest function in the Device Mapper core layer.

[0081] The `dmmakerequest` function in the core layer of the Device Mapper performs the validity verification of the original bio, obtains the multipath target driver instance corresponding to the mpatha device, performs a locking operation on the target driver instance, and then triggers the pre-map event notification chain, executing the pre-map hook callback function registered in this application. When the pre-map hook callback function is executed, it first obtains the pointer to the parent bio structure corresponding to the original bio from the input parameters, extracts the unique identifier of the parent IO request, allocates memory space from the slab cache pool, creates a DMIO tracing context instance, initializes each field in the instance, writes the unique identifier of the parent IO request into the parent IO global unique identifier field, writes the major and minor device numbers of the mpatha device into the corresponding fields, writes the type identifier of the multipath target driver into the target driver type field, finds the corresponding parent IO tracing structure instance through the unique identifier of the parent IO request, writes the pointer of the instance into the parent IO tracing structure pointer field, initializes the head node of the child bio tracing entry list, and initializes the atomic reference count of the DMIO tracing context instance to 1 through the `krefinit` function. After initialization is complete, the pre-map hook callback function writes the pointer to the DMIO tracking context instance into the private data extension field of the dm_io structure corresponding to the original bio, thus completing the execution of the pre-map hook.

[0082] After the pre-map hook is executed, the Device Mapper core layer calls the map function of the multi-path target driver to process the parent IO request corresponding to the original bio. Based on the queue length path load balancing strategy, the multi-path target driver queries the current queue lengths of the two physical paths sda and sdb, clones the 32KB original bio into two independent 16KB sub-bios, where sub-bio1 corresponds to the sda ​​physical path and sub-bio2 corresponds to the sdb physical path. The two sub-bio structures have independent memory addresses and biendio completion callback functions, completing the cloning and splitting operation. The map function completes and returns, ready to submit the two sub-bios to the corresponding physical path block devices via the dmsubmitbio function.

[0083] After the `map` function completes and before the `dmsubmitbio` function executes, the Device Mapper core layer triggers the post-map event notification chain, executing the post-map hook callback function registered in this application. When the post-map hook callback function executes, it first extracts the pointer to the DMIO tracking context instance from the `dm_io` structure corresponding to the parent bio. It then iterates through the two child bios generated by the `map` function, obtaining the corresponding `dm_io` structure instance for each child bio. The pointer to the DMIO tracking context instance is written into the private data extension field of the `dm_io` structure corresponding to the child bio. After each child bio's pointer injection operation is completed, the atomic reference count of the DMIO tracking context instance is incremented by 1 using the `krefget` function. After both child bios are injected, the atomic reference count changes from 1 to 3. After all child bio pointers are injected, the post-map hook execution is complete. The Device Mapper core layer then calls the `dmsubmitbio` function, submitting the two child bios to the general block layer processing of the physical block devices corresponding to `sda` and `sdb`, respectively.

[0084] After the two sub-bios enter the general block layer, they sequentially pass through the submitbio execution node, blkthrottle rate-limiting execution node, rqqosthrottle scheduling execution node, and gettag tag acquisition execution node in the bio phase, and the request generation execution node, plug-in / plug-out execution node, IO scheduler scheduling execution node, and hctx hardware queue acquisition execution node in the request phase. The host bus adapter card queue for the sdb physical path corresponding to sub-bio2 is idle, and sub-bio2 successfully passes through all preset IO blocking nodes without waiting or blocking, finally submitting to the device driver layer to complete IO processing, with a total time of 0.8ms. The host bus adapter card hardware queue for the sda ​​physical path corresponding to sub-bio1 is full. Sub-bio1 cannot obtain an available queue tag at the gettag tag acquisition execution node and enters a waiting state. After waiting 120ms, it obtains an available tag and continues the subsequent processing flow, finally completing IO processing, with a total time of 121.2ms.

[0085] During the process of the two sub-bios passing through each preset IO blocking node, the acquisition and processing function at the node will be executed. The acquisition and processing function will first obtain the dm_io structure corresponding to the sub-bio, extract the pointer of the DMIO tracking context instance from the private data extension field of the dm_io structure, read the pointer of the parent IO tracking structure from the instance, obtain the memory address of the parent IO tracking structure instance, calculate the residence time data of the sub-bio at the current preset IO blocking node, and combine the physical block device number, read / write type, unique identifier and residence time data corresponding to the sub-bio to form a sub-IO statistics entry, and write it into the sub-IO statistics chain table of the parent IO tracking structure.

[0086] In alternative implementations, such as Figure 9 As shown, after the atomic reference count of the initialization DMIO tracking context is 1, the process includes: S170: When the sub-bio completes IO processing, the sub-IO completion callback of the Device Mapper core layer is triggered, and the atomic reference count of the corresponding DMIO tracking context is decremented by 1; S180: When the parent bio completes IO processing, the parent IO completion callback of the Device Mapper core layer is triggered, and the atomic reference count of the corresponding DMIO tracking context is decremented by 1; S190: When the atomic reference count of the DMIO tracing context returns to 0, release the memory space corresponding to the DMIO tracing context.

[0087] Specifically, after sub-bio2 completes its I / O processing, it triggers the sub-IO completion callback function in the Device Mapper core layer. During the callback function's execution, the atomic reference count of the DMIO tracking context instance is decremented by 1 using the `krefput` function, changing the count from 3 to 2. After sub-bio1 completes its I / O processing, it similarly triggers its sub-IO completion callback function, decrementing the atomic reference count by 1 using the `krefput` function, changing the count from 2 to 1. Once both sub-bios are complete, the Device Mapper core layer triggers the parent I / O completion callback function corresponding to the parent bio. During the callback function's execution, the atomic reference count is decremented by 1 using the `krefput` function, changing the count from 1 to 0. After the atomic reference count reaches 0, the kernel automatically calls the preset release function to release the memory space occupied by the DMIO tracking context instance, completing the entire I / O request tracing process.

[0088] The user-space monitoring process reads the full-link tracing data of the IO request through the preset file nodes of debugfs, obtaining the total time of the original parent IO and detailed time data of the two child bios. Child bio2's total time was 0.8ms, with no abnormal time consumption at any of the preset IO blocking nodes. Child bio1's total time was 121.2ms, with the gettag acquisition execution node's dwell time being 120ms, corresponding to the physical block device sda. Based on the read data, the monitoring process outputs the cause of the IO request's latency anomaly: a waiting block caused by the host bus adapter card queue being full on the sda ​​physical path, occurring at the gettag acquisition execution node in the general block layer.

[0089] Those skilled in the art will understand that the technical solutions disclosed in this application can be applied not only to the multi-path scenarios in the above embodiments, but also to various target driver scenarios such as dmcrypt, dmverity, dmthin, and dmraid, as well as scenarios with multi-layer Device Mapper device stacking. The scope of protection of this application is not limited to specific target driver types.

[0090] This application also discloses a Linux kernel I / O event handling exception detection system, such as... Figure 10 As shown, it includes: The time consumption acquisition module 01 is used to collect the residence time of each IO request at the corresponding IO blocking node by the percpu variable at the preset IO blocking node of the Linux kernel storage stack. The preset IO blocking node is the execution node that causes the IO request to be blocked during the bio stage and request stage of the Linux kernel storage stack. Data output module 02 is used to create a preset file node in the kernel debugfs file system. When a user space reads the file node, it determines the total resident time in the percpu variable corresponding to all CPU cores and outputs the total resident time to the user space through the file node. The anomaly monitoring module 03 is used to create a corresponding IO trace structure for each IO stage of each block device in the Linux kernel. When an IO request enters the corresponding IO stage, a structure instance based on the IO trace structure is generated and added to the corresponding trace list. When the IO request leaves the corresponding IO stage, the instance is removed from the list. Based on the total dwell time and the structure instance, the module monitors in real time whether the IO request processing is abnormal.

[0091] It should be noted that the preset thresholds mentioned in the embodiments of this application, including the IO dwell time threshold and the in-transit IO existence time threshold, can be configured according to the actual business scenario and system performance requirements. This application does not limit the specific values ​​of these thresholds. The user-mode monitoring process can be implemented in various forms, including as a standalone daemon process, integrated into existing system monitoring tools, or provided to users as a command-line tool. This application does not limit the specific implementation form of the user-mode monitoring process.

[0092] Based on the same principle, the implementation of the Linux kernel IO event handling anomaly detection system in this application can refer to the implementation of the above method, and will not be described in detail here.

[0093] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for detecting anomalies in Linux kernel I / O event handling, characterized in that, include: At the preset IO blocking node in the Linux kernel storage stack, the dwell time of each IO request at the corresponding IO blocking node is collected by the percpu variable. The preset IO blocking node is the execution node that causes IO request blocking during the bio phase and request phase of the Linux kernel storage stack. A preset file node is created in the kernel debugfs file system. When a read request for the file node is received from user space, the total resident time in the percpu variable corresponding to all CPU cores is determined, and the total resident time is output to user space through the file node. For each I / O stage of each block device in the Linux kernel, a corresponding I / O trace structure is created. When an I / O request enters the corresponding I / O stage, a structure instance based on the I / O trace structure is generated and added to the corresponding trace list. When the I / O request leaves the corresponding I / O stage, the instance is removed from the list. The I / O request processing is monitored in real time for abnormalities based on the total dwell time and the structure instance.

2. The Linux kernel I / O event handling anomaly detection method according to claim 1, characterized in that, The process of collecting the dwell time of each IO request at the corresponding IO blocking node at the preset IO blocking node in the Linux kernel storage stack through the percpu variable includes: Determine the CPU core number currently executing the I / O request, and obtain the memory address of the percpu variable corresponding to that CPU core; Write the timestamps of the I / O request entering the node and leaving the node to the memory address; The dwell time of the IO request on the corresponding node is calculated based on the timestamp of the same IO request in the same percpu variable.

3. The Linux kernel I / O event handling anomaly detection method according to claim 1, characterized in that, The method further includes: A lock-free circular buffer is created in the kernel for each block device. This lock-free circular buffer is a memory buffer shared by kernel mode and user mode. When an IO request leaves the corresponding IO stage, the IO leave event of that IO request is written to the lock-free circular buffer; The IO event data in the lockless circular buffer is output to the user space through the io_uring interface, which is an asynchronous IO interface provided by the Linux kernel.

4. The Linux kernel I / O event handling anomaly detection method according to claim 1, characterized in that, The method further includes: Register pre-map and post-map hooks in the core layer of the Linux kernel Device Mapper framework. The pre-map hook is a pre-callback function attached to the Device Mapper IO processing flow, and the post-map hook is a post-callback function attached to the Device Mapper IO processing flow. The execution timing of the pre-map hook is set to be before the Device Mapper core layer calls the target-driven map function; The execution timing of the post-map hook is set to be after the target driver's map function has been executed and before the dm_submit_bio function has been executed. The target driver is the driver module in the Device Mapper framework that implements the specific IO processing logic.

5. The Linux kernel I / O event handling anomaly detection method according to claim 4, characterized in that, The setting of the execution timing of the pre-map hook to be before the Device Mapper core layer calls the target-driven map function includes: When the Device Mapper core layer receives an IO request, the pre-map hook is triggered to execute; Create a DMIO tracing context for the parent bio corresponding to the IO request, and store the DMIO tracing context in the dm_io structure of the DeviceMapper core layer. The DMIO tracing context is a structure that stores the parent IO tracing information, and the dm_io structure is a private IO request structure natively managed by the DeviceMapper core layer. The atomic reference count of the DMIO tracing context is initialized to 1. The atomic reference count is a thread-safe reference count provided by the Linux kernel kref mechanism.

6. The Linux kernel I / O event handling anomaly detection method according to claim 5, characterized in that, The setting of the post-map hook to execute at the point after the target-driven map function has finished executing and before the dm_submit_bio function has executed includes: After the target-driven map function completes execution, the post-map hook is triggered. Iterate through all the subbios generated by the map function, allocate a corresponding dm_io structure for each subbio, and inject the DMIO tracking context pointer corresponding to the parent bio into the dm_io structure of each subbio; After each pointer injection operation of a subbio is completed, the atomic reference count of the corresponding DMIO tracking context is incremented by 1.

7. The Linux kernel I / O event handling anomaly detection method according to claim 6, characterized in that, The process of collecting the dwell time of each IO request at the corresponding IO blocking node at the preset IO blocking node in the Linux kernel storage stack through the percpu variable includes: When a subbio enters a preset IO blocking node, the DMIO tracking context pointer is extracted from the dm_io structure corresponding to that subbio; Obtain the pointer to the parent IO trace structure in the DMIO trace context; Write the dwell time data of the subbio at the corresponding preset IO blocking node into the subIO statistics item of the parent IO tracking structure.

8. The Linux kernel I / O event handling anomaly detection method according to claim 5, characterized in that, After initializing the atomic reference count of the DMIO tracing context to 1, the process includes: When the subbio completes the IO processing, the subIO completion callback of the Device Mapper core layer is triggered, and the atomic reference count of the corresponding DMIO tracking context is decremented by 1; When the parent bio completes the IO processing, the parent IO completion callback of the Device Mapper core layer is triggered, and the atomic reference count of the corresponding DMIO tracking context is decremented by 1. When the atomic reference count of the DMIO tracing context reaches 0, the memory space corresponding to the DMIO tracing context is released.

9. The Linux kernel I / O event handling anomaly detection method according to claim 4, characterized in that, The registration of pre-map and post-map hooks in the core layer of the Linux kernel Device Mapper framework includes: When a subbio enters the IO processing flow of the next-layer Device Mapper device, the pre-map hook of the corresponding next-layer Device Mapper device is triggered to create a DMIO tracing context of the corresponding layer for the subbio. After the map function corresponding to the subbio is executed, the post-map hook of the next-level Device Mapper is triggered to inject the corresponding level of DMIO tracing context pointer into the next-level subbio generated by the subbio split from the subbio.

10. A Linux kernel I / O event handling anomaly detection system, characterized in that, include: The time consumption acquisition module is used to collect the residence time of each IO request at the corresponding IO blocking node by using the percpu variable at the preset IO blocking node of the Linux kernel storage stack. The preset IO blocking node is the execution node that causes IO request blocking during the bio phase and request phase of the Linux kernel storage stack. The data output module is used to create a preset file node in the kernel debugfs file system. When it receives a read request for the file node from the user space, it determines the total resident time in the percpu variable corresponding to all CPU cores and outputs the total resident time to the user space through the file node. The anomaly monitoring module is used to create a corresponding IO trace structure for each IO stage of each block device in the Linux kernel. When an IO request enters the corresponding IO stage, a structure instance based on the IO trace structure is generated and added to the corresponding trace list. When the IO request leaves the corresponding IO stage, the instance is removed from the list. The module monitors whether the IO request processing is abnormal in real time based on the total dwell time and the structure instance.

Images