Method and system for collecting and processing full-link performance data of NVMe / TCP protocol

Abstract

This application provides a method and system for collecting and processing NVMe / TCP protocol end-to-end performance data. The method includes: a user-space control module parsing a configuration file to generate monitoring configuration parameters, and sending the monitoring configuration parameters to a kernel-space monitoring module through a character device interface; the kernel-space monitoring module registering kernel tracepoint hooks corresponding to the NVMe / TCP protocol end-to-end, collecting end-to-end event data of the corresponding I / O stream through the tracepoint hooks, writing the collected event data into a pre-allocated per-CPU circular buffer, performing anomaly detection based on the event data, and generating anomaly events; the kernel-space monitoring module exporting statistical data and anomaly events to the user-space control module through a virtual file system, whereby the user-space control module performs anomaly root cause localization. This application reduces the overhead of monitoring operations while achieving end-to-end performance collection and anomaly diagnosis.

Description

Technical Field

[0001] This invention relates to the field of computer storage system performance monitoring technology, and in particular to a method and system for acquiring and processing NVMe / TCP protocol full-link performance data. Background Technology

[0002] With the widespread application of NVMe / TCP protocols in distributed storage systems, existing monitoring solutions have significant shortcomings. Application-layer tracing solutions cannot reach the depths of the kernel protocol stack, cannot obtain performance event data across the entire NVMe / TCP protocol chain, and cannot pinpoint the root cause of anomalies. General kernel observation solutions can only collect discrete data from a single subsystem, cannot establish end-to-end I / O chain correlations, and have long data collection and analysis processes, resulting in high CPU overhead during normal operation. They can also preempt kernel resources for NVMe / TCP protocol services, affecting the low-latency characteristics of service I / O, and cannot operate stably in production environments for extended periods. Summary of the Invention

[0003] This application provides a method for collecting and processing NVMe / TCP protocol end-to-end performance data. While collecting end-to-end performance events and diagnosing anomalies in the NVMe / TCP protocol end-to-end chain, it executes kernel interrupt context constraints to prevent monitoring operations from affecting the normal operation of service I / O. Another aspect of this application provides a system for collecting and processing NVMe / TCP protocol end-to-end performance data.

[0004] This application discloses a method for collecting and processing NVMe / TCP protocol end-to-end performance data, the method comprising: The user-mode control module parses the configuration file to generate monitoring configuration parameters, and then sends the monitoring configuration parameters to the kernel-mode monitoring module through the character device interface to complete the initialization of the monitoring environment. The kernel-mode monitoring module registers kernel tracepoint hooks corresponding to the entire NVMe / TCP protocol link. It collects the entire link event data of the corresponding I / O stream through the tracepoint hooks, writes the collected event data into a pre-allocated per-CPU circular buffer, performs anomaly detection based on the event data, and generates abnormal events. The kernel-mode monitoring module exports statistical data and abnormal events to the user-mode control module via the virtual file system, whereby the user-mode control module locates the root cause of the abnormality.

[0005] Optionally, the collection of end-to-end event data for the corresponding I / O stream via the tracepoint hook includes: Block layer event data of I / O requests is collected through the block device layer tracepoint hook, and NVMe command event data of I / O requests is collected through the NVMe driver layer tracepoint hook. Collect TCP transmission event data of I / O requests through NVMe / TCP protocol layer tracepoint hooks; Collect backend processing event data for I / O requests through the tracepoint hook on the Target side.

[0006] Optionally, the collection of block layer event data for I / O requests via the block device layer tracepoint hook includes: When the tracepoint hook is triggered, perform fast path verification to match the I / O requests corresponding to the monitoring configuration parameters; Assign a predefined record structure to the matching I / O request, recording the timestamp of the event trigger and the event type; Write the record structure to the per-CPU circular buffer of the corresponding CPU and update the statistics counter.

[0007] Optionally, the step of performing anomaly detection based on event data and generating anomaly events includes: Based on the collected end-to-end event data, a comprehensive suspicion value is calculated for each I / O flow. The comprehensive suspicion value is used to quantify the abnormal risk level of the I / O flow. The metadata record depth level of the corresponding I / O stream is obtained by mapping the comprehensive skepticism value; Based on the metadata record depth level, update the execution logic of the tracepoint hook for the corresponding I / O stream and adjust the collection dimensions of subsequent event data; Based on the event data obtained after adjusting the collection dimensions, anomaly detection is completed and abnormal events are generated.

[0008] Optionally, calculating the overall skepticism value for each I / O flow includes: Calculate the latency deviation value of the I / O flow based on I / O flow event data within a fixed statistical period; Calculate the trend deterioration value of I / O flow based on I / O flow delay change data within a continuous statistical period; Based on I / O stream error event data within the statistical period, calculate the abnormal event triggering degree value of the I / O stream; Based on the preset weights, the latency deviation value, the trend deterioration value, and the abnormal event triggering value are combined to obtain the comprehensive suspicion value of the I / O flow.

[0009] Optionally, the calculation of the delay deviation value of the I / O flow includes: The latency dynamic baseline of I / O flow is updated based on the exponential moving average algorithm; The relative delay deviation rate is calculated based on the difference between the measured average delay of the I / O flow and the dynamic delay baseline within the statistical period; the delay deviation value of the I / O flow is generated based on the relative delay deviation rate and the preset deviation threshold.

[0010] Optionally, updating the tracepoint hook execution logic of the corresponding I / O stream based on the metadata record depth level and adjusting the collection dimensions of subsequent event data includes: The overall skepticism value is mapped to discrete metadata record depth levels, and each metadata record depth level corresponds to a fixed set of event data collection dimensions; Metadata record depth levels are synchronized to the corresponding I / O stream's tracepoint hook execution context using atomic variables; When the tracepoint hook is triggered, the metadata record depth level stored in the atomic variable is read, and the event data writing operation in the corresponding collection dimension set is executed.

[0011] Optionally, the step of performing the event data writing operation within the corresponding collection dimension set includes: Match the collection dimension set corresponding to the depth level of the metadata record and write the basic metadata to the pre-allocated record structure; Determine whether the collection dimension set contains extended metadata dimensions. If it does, read the event parameters of the corresponding kernel subsystem and write the extended metadata to the pre-allocated record structure. After writing event data for all matching dimensions, the record structure is written to the per-CPU circular buffer.

[0012] Optionally, after completing anomaly detection and generating anomaly events based on the event data obtained after adjusting the collection dimensions, the method further includes: Based on the event data obtained after adjusting the collection dimensions, the root cause of the anomaly is located and the comprehensive suspicion value of the corresponding I / O stream is updated. If the overall skepticism value of the I / O flow is lower than the preset threshold within a continuous statistical period, the metadata record depth level of the corresponding I / O flow will be reset to the initial level, and the collection of extended metadata dimensions will be stopped.

[0013] This application also discloses an NVMe / TCP protocol end-to-end performance data acquisition and processing system, the system comprising: The user-mode control module parses the configuration file to generate monitoring configuration parameters, and then sends these parameters to the kernel-mode monitoring module via the character device interface to complete the initialization of the monitoring environment. The kernel-mode monitoring module registers kernel tracepoint hooks corresponding to the entire NVMe / TCP protocol link. It collects the entire link event data of the corresponding I / O stream through the tracepoint hooks, writes the collected event data into a pre-allocated per-CPU circular buffer, performs anomaly detection based on the event data, and generates abnormal events. The monitoring and reporting module, which is part of the kernel-mode monitoring module, exports statistical data and abnormal events to the user-mode control module via the virtual file system. The user-mode control module then locates the root cause of the abnormalities.

[0014] As described in the above technical solution, the user-space control module sends monitoring configuration parameters to the kernel-space monitoring module for initialization. The kernel-space monitoring module collects end-to-end event data of the I / O stream by registering end-to-end tracepoint hooks, writes it to a pre-allocated per-CPU circular buffer, performs anomaly detection in kernel space based on the collected event data, and finally exports the results to user space to generate a report. Throughout this process, tracepoint hooks directly collect end-to-end event data in kernel space without cross-layer data copying; the per-CPU circular buffer avoids multi-core contention; and in-situ anomaly detection in kernel space shortens the data flow path. These technical features work together to achieve end-to-end performance collection and anomaly diagnosis of the NVMe / TCP protocol while keeping the system overhead of monitoring operations extremely low, thus avoiding impact on business I / O operations. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the NVMe / TCP protocol end-to-end performance data acquisition and processing method in the embodiments of this application; Figure 2 This is a flowchart illustrating the process of collecting full-link event data of the corresponding I / O stream through tracepoint hooks in the NVMe / TCP protocol full-link performance data collection and processing method in this application embodiment; Figure 3 This is a flowchart illustrating the process of collecting block layer event data of I / O requests through the block device layer tracepoint hook in the NVMe / TCP protocol full-link performance data collection and processing method in this application embodiment; Figure 4This is a flowchart illustrating the process of anomaly detection and generation of abnormal events based on event data in the NVMe / TCP protocol full-link performance data acquisition and processing method in this application embodiment. Figure 5 This is a flowchart illustrating the method for collecting and processing NVMe / TCP protocol full-link performance data in this application embodiment, which calculates the comprehensive skepticism value for each I / O stream. Figure 6 This is a flowchart illustrating the calculation of the latency deviation value of the I / O stream in the NVMe / TCP protocol end-to-end performance data acquisition and processing method in this application embodiment; Figure 7 This is a flowchart illustrating the process of the NVMe / TCP protocol end-to-end performance data collection and processing method in this application embodiment, which updates the tracepoint hook execution logic of the corresponding I / O stream according to the metadata record depth level and adjusts the collection dimensions of subsequent event data. Figure 8 This is a schematic diagram of the event data writing operation process within the corresponding collection dimension set of the NVMe / TCP protocol full-link performance data collection and processing method in the embodiments of this application; Figure 9 This is a flowchart illustrating the process of collecting and processing NVMe / TCP protocol full-link performance data in the embodiments of this application, based on event data obtained after adjusting the collection dimensions, completing anomaly detection and generating abnormal events. Figure 10 This is a schematic diagram of the structure of the NVMe / TCP protocol end-to-end performance data acquisition and processing system in the embodiments of this application. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not limiting, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without such specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0018] In this embodiment, taking the NVMe / TCP protocol performance monitoring scenario of a distributed storage cluster as an example, existing technologies use user-space instrumentation to collect performance data. The user-space agent program reads statistical data from the kernel network stack and block device layer through system calls. Existing solutions can only obtain global statistical data from the kernel subsystem and cannot correlate end-to-end events of a single I / O stream. When I / O latency increases, only the latency value is known, but the specific level and root cause of the latency cannot be located. Furthermore, frequent system calls executed by the user-space agent program consume significant CPU resources. In high-concurrency I / O scenarios, this can preempt CPU time slices from business processes, causing fluctuations in business I / O latency and making continuous operation in a production environment impossible.

[0019] Existing metadata collection solutions for NVMe / TCP protocol performance monitoring all employ a globally unified collection strategy. The global full-data collection solution enables full-dimensional metadata collection for all I / O streams. While this obtains complete diagnostic data, it causes the number of executed instructions for tracepoint hooks to exceed the hard constraints of the kernel interrupt context, resulting in excessively high normal CPU overhead and severely impacting business operations. The global minimalist collection solution only collects basic metadata for all I / O streams. While it can control system overhead, it cannot obtain multi-dimensional extended metadata to support anomaly root cause localization; it can only identify anomalies but cannot perform root cause diagnosis. Therefore, how to obtain sufficient full-dimensional metadata to support NVMe / TCP protocol anomaly root cause localization while strictly meeting the hard constraints of the number of instructions and memory write overhead in the interrupt context where the kernel tracepoint hook resides, and keeping the normal operating overhead of the monitoring system at an extremely low level, has become a significant challenge.

[0020] To address at least one of the problems existing in the prior art, this embodiment discloses a method for collecting and processing NVMe / TCP protocol end-to-end performance data, such as... Figure 1 As shown, the method includes: S100: The user-mode control module parses the configuration file to generate monitoring configuration parameters, and sends the monitoring configuration parameters to the kernel-mode monitoring module through the character device interface to complete the initialization of the monitoring environment.

[0021] Specifically, the user-space control module sends monitoring configuration parameters to the kernel-space monitoring module for initialization. The kernel-space monitoring module collects end-to-end event data of the I / O flow by registering end-to-end tracepoint hooks, writes it to a pre-allocated per-CPU circular buffer, performs anomaly detection in kernel space based on the collected event data, and finally exports the results to user space to generate a report. The tracepoint hooks in this application directly complete the correlation collection of end-to-end events in kernel space, eliminating the need for cross-layer data copying. The per-CPU circular buffer avoids multi-core contention, and the in-situ anomaly detection in kernel space shortens the data flow path. While achieving end-to-end performance collection and anomaly diagnosis of the NVMe / TCP protocol, the system overhead of monitoring operations is kept to a very low level, avoiding any impact on business I / O operations.

[0022] In a specific example, the startup process of the user-space control module is as follows: the user starts the executable program of the user-space control module through the command-line interface, passing in the storage path of the configuration file. The user-space control module reads the contents of the configuration file, parses and validates the fields in the configuration file, and generates structured monitoring configuration parameters. The configuration file is stored in INI format and includes fields such as monitoring switches, statistical periods, tracepoint hook types, I / O stream filtering rules, anomaly detection thresholds, and export paths. After the user-space control module completes the configuration file parsing, it opens the character device file created by the operating system kernel and copies the monitoring configuration parameters from user space to kernel space through the ioctl system call, and then distributes them to the kernel-space monitoring module.

[0023] Specifically, the character device interface is created by the kernel-mode monitoring module during the initialization phase. The major device number of the character device is dynamically allocated by the operating system, while the minor device number is statically defined by the kernel-mode monitoring module. The file operation structure corresponding to the character device implements file operation functions such as ioctl, open, release, and read to support communication between the user-mode control module and the kernel-mode monitoring module. After receiving the monitoring configuration parameters from the user-mode control module, the kernel-mode monitoring module stores the monitoring configuration parameters in a global structure in the kernel space, completing the initialization operation of the monitoring environment. The initialization operation includes memory allocation for the per-CPU ring buffer, binding of the registration callback function for the tracepoint hook, creation of the kernel work queue, and initialization of global statistical variables. It should be noted that the performance monitoring method provided in this application embodiment runs in a Linux operating system environment and is adapted to the storage cluster scenario of the NVMe over TCP protocol specification. In this application embodiment, the user-mode control module runs in the user space of the operating system and is deployed as an executable program on the host node and target node of the storage cluster. The kernel-mode monitoring module is deployed as a loadable kernel module in the operating system kernel of the corresponding node.

[0024] S200: The kernel-mode monitoring module registers the kernel tracepoint hook corresponding to the entire NVMe / TCP protocol link. It collects the entire link event data of the corresponding I / O stream through the tracepoint hook, writes the collected event data into the pre-allocated per-CPU circular buffer, completes anomaly detection based on the event data, and generates anomaly events.

[0025] S300: The kernel-mode monitoring module exports statistical data and abnormal events to the user-mode control module through the virtual file system, whereby the user-mode control module locates the root cause of the abnormality.

[0026] In this embodiment, after the kernel-mode monitoring module completes initialization, it registers the kernel tracepoint hooks corresponding to the entire NVMe / TCP protocol link according to the tracepoint hook type specified in the monitoring configuration parameters. A tracepoint is a static probe mechanism natively provided by the Linux kernel. Tracepoint hooks are predefined in the kernel source code at the code path. When the kernel executes to the corresponding code path, the tracepoint hook is triggered, and the registered callback function is executed. The execution context of the tracepoint hook is an interrupt context. During execution, sleeping, blocking, dynamic memory allocation, and other operations are prohibited. The execution time needs to be controlled within nanoseconds to avoid affecting the normal execution of the native kernel code path.

[0027] For example, the kernel tracepoint hooks corresponding to the entire NVMe / TCP protocol chain cover the kernel code paths of the block device layer, NVMe driver layer, NVMe / TCP protocol layer, TCP network layer, and target-side processing layer. The kernel-mode monitoring module binds a corresponding callback function to each tracepoint hook that needs to be registered. The callback function implements the collection and writing operations of event data. When the kernel executes to the code path of the corresponding tracepoint hook, it triggers the execution of the callback function, collects the entire chain event data of the current I / O stream, and writes the collected event data to a pre-allocated per-CPU circular buffer.

[0028] Specifically, the per-CPU circular buffer is a fixed-size circular buffer allocated separately to each CPU. Each CPU can only access its own corresponding circular buffer. The buffers of different CPUs are independent of each other, and there is no contention for multi-core concurrent access. Data writing and reading operations can be completed without locking. The memory of the per-CPU circular buffer is pre-allocated by the kernel-mode monitoring module during the initialization phase through a kernel-specific memory allocation interface. The memory size can be adjusted through monitoring configuration parameters. The buffer adopts a producer-consumer model to implement data writing and reading. The producer is the callback function of the tracepoint hook, and the consumer is the kernel work queue of the kernel-mode monitoring module. Those skilled in the art will understand that the per-CPU variable used in the embodiments of this application is a variable type designed by the Linux kernel specifically for multi-core concurrent scenarios. Its principle is to create an independent copy of the variable for each CPU. Each CPU only accesses its own corresponding copy. The copies of different CPUs are independent of each other, and there is no contention for multi-core concurrent access. Therefore, read and write operations can be completed without locking, avoiding the cache line thrashing problem caused by multi-core contention, and significantly improving access efficiency. The memory allocation of per-CPU variables is completed during the initialization phase. There are no dynamic memory operations during operation, which fully adapts to the execution constraints of the interrupt context. This is the basis for realizing lock-free data acquisition in the embodiments of this application.

[0029] In this embodiment, the kernel-mode monitoring module periodically reads collected event data from the per-CPU circular buffer via a kernel work queue, performs anomaly detection based on the event data, and generates anomaly events. The kernel work queue runs in the kernel process context, is not subject to interrupt context constraints, can perform complex computational operations, and does not affect the execution of kernel code paths for business I / O. During anomaly detection, the kernel-mode monitoring module compares the collected event data with the anomaly detection threshold in the monitoring configuration parameters, identifies anomaly indicators exceeding the threshold, analyzes the anomaly indicators using a multi-algorithm fusion approach, determines the anomaly type, and generates corresponding anomaly events. Anomaly events include fields such as the timestamp of the anomaly occurrence, the corresponding I / O stream identifier, the anomaly indicator value, and the anomaly type. In a specific example, after completing anomaly detection, the kernel-mode monitoring module aggregates the collected event data to generate statistical data, which includes indicators such as average latency, IOPS, bandwidth, error rate, and queue depth for each I / O stream. The kernel-mode monitoring module writes the statistical data and anomaly events into the corresponding file nodes of the virtual file system. The virtual file system is implemented using the proc file system, and the file node path can be specified through monitoring configuration parameters. The user-space control module reads statistical data and abnormal events from the proc file system using the read system call. After formatting the data, it generates a visual monitoring report, which can be stored in text or JSON format.

[0030] Furthermore, those skilled in the art will understand that collecting end-to-end event data of the corresponding I / O stream through tracepoint hooks involves four levels of collection operations, which correspond to the block device layer, NVMe driver layer, NVMe / TCP protocol layer, and the kernel code path of the target end, respectively.

[0031] Specifically, block-level event data for I / O requests is collected through tracepoint hooks in the block device layer. The block device layer is a subsystem in the Linux kernel that manages block storage devices; all I / O requests to storage devices must be scheduled and processed by the block device layer. The tracepoint hooks corresponding to the block device layer include `block_rq_insert`, `block_rq_issue`, and `block_rq_complete`, which correspond to the kernel code paths where an I / O request is inserted into the scheduling queue, an I / O request is sent to the driver, and an I / O request is completed, respectively. The kernel-mode monitoring module binds corresponding callback functions to these tracepoint hooks. When the kernel executes to the corresponding code path, the callback function is triggered, collecting block-level event data for the I / O request. In a specific example, the `block_rq_insert` hook callback function collects block-level event data including the starting sector of the I / O request, data length, I / O priority, scheduler type, and timestamp of insertion into the queue. The `block_rq_issue` hook callback function collects block-level event data including the timestamp of the I / O request being sent to the driver, the device number corresponding to the request, and the queue identifier. The callback function of the block_rq_complete hook collects block-level event data including the timestamp of I / O request completion, processing result, error code, number of bytes transferred, etc.

[0032] Specifically, NVMe command event data for I / O requests is collected through tracepoint hooks in the NVMe driver layer. The NVMe driver layer is the driver subsystem in the Linux kernel that interfaces with NVMe storage devices. It is responsible for encapsulating block device layer I / O requests into NVMe protocol commands, sending them to the NVMe device, and processing the completion responses returned by the NVMe device. The tracepoint hooks corresponding to the NVMe driver layer include `nvme_setup_cmd`, `nvme_complete_rq`, and `nvme_sq_full`, which correspond to the kernel code paths for NVMe command encapsulation, NVMe request completion, and a full commit queue, respectively. The kernel-mode monitoring module binds corresponding callback functions to these tracepoint hooks to collect NVMe command event data for I / O requests. In a specific example, the callback function of the `nvme_setup_cmd` hook collects NVMe command event data including the NVMe command's opcode, command identifier, namespace identifier, logical block address, and data length. The callback function of the `nvme_complete_rq` hook collects NVMe command event data including the command completion timestamp, status code, and error type. The event data collected by the callback function of the nvme_sq_full hook includes the timestamp of when the queue is full, the queue identifier, and the queue depth.

[0033] Specifically, tracepoint hooks in the NVMe / TCP protocol layer collect TCP transmission event data for I / O requests. The NVMe / TCP protocol layer is a kernel subsystem in the Linux kernel that implements the NVMeoverTCP protocol. It is responsible for encapsulating NVMe commands into TCP protocol packets, transmitting them over the network to the target node, and processing the TCP packets returned by the target node to parse the NVMe response. The tracepoint hooks corresponding to the NVMe / TCP protocol layer include `nvme_tcp_queue_rq`, `nvme_tcp_done_send_req`, and `nvme_tcp_recv_skb`, which correspond to the kernel code paths where an NVMe request enters the TCP send queue, an NVMe request is sent, and an NVMe response packet is received, respectively. The kernel-mode monitoring module binds corresponding callback functions to these tracepoint hooks to collect TCP transmission event data for I / O requests. In this specific example, the TCP transmission event data collected by the callback function of the `nvme_tcp_queue_rq` hook includes the TCP socket address corresponding to the request, queue identifier, PDU type, and data length. The callback function of the `nvme_tcp_done_send_req` hook collects event data including the timestamp of the request completion, the number of bytes sent, and the socket status. The callback function of the `nvme_tcp_recv_skb` hook collects event data including the timestamp of the received data packet, the data length, and the PDU parsing result.

[0034] Specifically, backend processing event data for I / O requests is collected through tracepoint hooks on the target side. The target side is the storage node in the NVMe / TCP protocol that receives and processes host-side I / O requests. The corresponding tracepoint hooks on the target side include `nvmet_tcp_try_recv_pdu`, `nvmet_tcp_send_response`, and `nvmet_req_execute`, which correspond to the kernel code paths for the target side receiving PDUs, sending responses, and executing I / O requests, respectively. The kernel-mode monitoring module binds corresponding callback functions to these tracepoint hooks to collect backend processing event data for I / O requests. In a specific example, the callback function of the `nvmet_tcp_try_recv_pdu` hook collects backend processing event data including the timestamp of PDU reception, PDU type, data length, and parsing result. The callback function of the `nvmet_req_execute` hook collects event data including the timestamp of I / O request execution, command type, and namespace information. The callback function of the nvmet_tcp_send_response hook collects event data including the timestamp of the response being sent, the data length, and the status code.

[0035] It should be noted that the interrupt context involved in this application embodiment is the execution environment in the Linux kernel for handling hardware and software interrupts, which has strict execution constraints. The execution priority of the interrupt context is higher than that of the process context. Sleeping, scheduling, and blocking operations cannot occur during execution, otherwise it will lead to kernel deadlock. Dynamic memory allocation cannot be performed during execution, as it may trigger sleep operations. The execution duration must be strictly controlled within nanoseconds to microseconds; otherwise, it will block the processing of subsequent interrupts, causing hardware device response timeouts and triggering service failures. In the tracepoint hook callback function of this application embodiment, all execution logic strictly follows the execution constraints of the interrupt context, with no sleeping, no blocking, and no dynamic memory allocation. The execution duration is strictly controlled within the hard constraint range and will not affect the normal operation of service I / O.

[0036] In alternative implementations, such as Figure 2 As shown, the collection of end-to-end event data for the corresponding I / O stream via tracepoint hooks includes: S210: Collect block layer event data of I / O requests through the block device layer tracepoint hook, and collect NVMe command event data of I / O requests through the NVMe driver layer tracepoint hook.

[0037] S220: Collects TCP transmission event data of I / O requests through the NVMe / TCP protocol layer tracepoint hook.

[0038] S230: Collect backend processing event data of I / O requests through the tracepoint hook on the Target side.

[0039] Specifically, when the tracepoint hook is triggered, a fast path check is performed to match the I / O requests corresponding to the monitoring configuration parameters. Fast path check filters out I / O requests that do not need monitoring, preventing invalid data collection operations from consuming CPU resources. Fast path check first reads the global monitoring switch variable. If the monitoring switch is off, it returns directly without any subsequent operations. If the monitoring switch is on, it reads the device number, namespace identifier, queue identifier, and other parameters corresponding to the current I / O request and compares them with the I / O stream filtering rules in the monitoring configuration parameters to determine if the current I / O request falls within the scope of monitoring. If the current I / O request does not meet the filtering rules, it returns directly without further operations. If it meets the filtering rules, it continues with subsequent steps. I / O stream filtering rules can be set through a configuration file, supporting whitelists or blacklists based on dimensions such as device number, namespace identifier, controller identifier, queue identifier, and IP address. Only I / O requests that meet the whitelist rules and do not meet the blacklist rules will be determined as I / O requests that need monitoring.

[0040] Specifically, a predefined record structure is allocated for each matched I / O request to record the timestamp and event type of the event. The record structure is a fixed-size structure predefined by the kernel-mode monitoring module to store collected event data. The memory layout of the record structure is determined during the compilation phase, and there are no dynamically expanded fields, avoiding dynamic memory allocation operations. The record structure contains basic fields such as timestamp, event type, I / O stream unique identifier, and command identifier, as well as multiple extended fields for storing event data at different levels. For each matched I / O request, the callback function retrieves a pre-allocated record structure instance from the per-CPU memory pool, writes the timestamp of the current event to the timestamp field of the record structure, writes the event type code corresponding to the current tracepoint hook to the event type field of the record structure, and simultaneously writes the unique identifier of the I / O stream, command identifier, and other basic information to the corresponding fields of the record structure.

[0041] In alternative implementations, such as Figure 3 As shown, the collection of block layer event data for I / O requests via the block device layer tracepoint hook includes: S211: Performs fast path verification when the tracepoint hook is triggered, matching the I / O requests corresponding to the monitoring configuration parameters.

[0042] S212: Allocate a predefined record structure for the matched I / O request, recording the timestamp of the event trigger and the event type.

[0043] S213: Write the record structure to the per-CPU circular buffer of the corresponding CPU and update the statistics counter.

[0044] In this embodiment, the timestamp is obtained through the kernel's native `ktime_get_real_ns` function, which returns the system's real-time time in nanoseconds, ensuring the accuracy of the event timing. The event type uses a predefined unsigned integer encoding, with each tracepoint hook corresponding to a unique event type code.

[0045] Specifically, the record structure is written to the per-CPU circular buffer of the corresponding CPU, and the statistics counter is updated. After the callback function completes the field filling of the record structure, it writes the record structure to the producer position of the per-CPU circular buffer corresponding to the current CPU, updates the producer pointer of the circular buffer, and completes the event data writing operation. At the same time, the callback function updates the global statistics counter of the kernel-mode monitoring module, incrementing the collection count of the corresponding event type by 1, completing the entire collection process. In this specific example, the statistics counter is implemented using the native atomic_t variables of the Linux kernel. Each event type corresponds to an independent atomic counter, and the addition and subtraction operations of atomic variables are single-CPU instruction-level atomic operations. The write operation of the per-CPU circular buffer is only executed in the interrupt context of the current CPU. There is no concurrent write from other CPUs, so the write can be completed without locking, ensuring the execution efficiency of the write operation.

[0046] The atomic operations used in this embodiment are a fundamental mechanism in the Linux kernel for multi-core concurrent synchronization. Their principle is to ensure the atomicity of memory read / write operations through lock prefix instructions provided by the CPU architecture. This means that during the operation, other CPUs cannot access the memory address, preventing dirty data issues caused by concurrent read / write operations. The execution of atomic operations is at the single-CPU instruction level, with extremely short execution time, no lock contention, no blocking, and fully adapts to the execution constraints of interrupt contexts. This forms the basis for achieving deep-level secure synchronization of metadata records in this embodiment.

[0047] In alternative implementations, such as Figure 4 As shown, the anomaly detection and generation of anomaly events based on event data includes: S240: Based on the collected end-to-end event data, calculate a comprehensive suspicion value for each I / O flow. The comprehensive suspicion value is used to quantify the abnormal risk level of the I / O flow.

[0048] S250: The metadata record depth level of the corresponding I / O stream is obtained by mapping the comprehensive skepticism value.

[0049] S260: Based on the metadata record depth level, update the tracepoint hook execution logic of the corresponding I / O stream and adjust the collection dimensions of subsequent event data.

[0050] S270: Based on the event data obtained after adjusting the collection dimensions, complete the anomaly detection and generate anomaly events.

[0051] In this embodiment, anomaly detection is performed based on event data to generate anomaly events. This process includes four execution steps, which form a complete closed-loop execution logic to achieve dynamic adaptation between the data collection dimensions and the anomaly risk.

[0052] Specifically, based on the collected end-to-end event data, a comprehensive suspicion score is calculated for each I / O flow. This score quantifies the anomaly risk level of the I / O flow. The kernel work queue of the kernel-mode monitoring module reads the collected end-to-end event data from the per-CPU circular buffer according to a fixed statistical period. The event data is aggregated according to the unique identifier of each I / O flow, generating performance metrics data for each I / O flow within the current statistical period. Based on the aggregated performance metrics data, the kernel work queue calculates a comprehensive suspicion score for each I / O flow. The comprehensive suspicion score is fixed between 0 and 1; a higher score indicates a higher anomaly risk level for the corresponding I / O flow and a stronger need for multi-dimensional diagnostic metadata. In this specific example, the duration of the fixed statistical period can be set through monitoring configuration parameters. The default setting is 100 milliseconds, and the duration can be adjusted according to business needs, with a minimum of 10 milliseconds and a maximum of 1 second. A shorter statistical period results in faster anomaly risk identification and response, but also higher computational overhead. The longer the statistical period, the lower the computational overhead, but the slower the response speed of anomaly identification. At the end of each statistical period, the kernel work queue triggers a calculation operation for the overall suspicion value, updating the overall suspicion value for all I / O flows within the monitoring range. It should be noted that the fixed duration of the statistical period and the corresponding specific value in this application can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the purpose of this invention is achieved; this application does not impose any limitations on this.

[0053] Specifically, the metadata record depth level for the corresponding I / O stream is obtained by mapping the overall suspicion value. The kernel-mode monitoring module predefines the mapping relationship between the overall suspicion value range and the metadata record depth level. Each metadata record depth level corresponds to a fixed overall suspicion value range. The kernel-mode monitoring module maps the corresponding metadata record depth level based on the range where the overall suspicion value of the I / O stream falls. The metadata record depth level is used to define the event data dimensions that need to be collected and written into the record structure in the tracepoint hook callback function. The higher the metadata record depth level, the richer the event data dimensions that need to be collected, and the more instructions the corresponding tracepoint hook callback function executes.

[0054] In this specific example, the metadata recording depth is divided into five levels: Level 0, Level 1, Level 2, Level 3, and Level 4. Each level corresponds to a different range of overall suspicion values ​​and a set of event data collection dimensions. Level 0 corresponds to a range where the overall suspicion value is below 0.1, and the collection dimension set only includes basic metadata. Level 1 corresponds to a range where the overall suspicion value is greater than or equal to 0.1 and less than 0.3, and the collection dimension set includes basic metadata and TCP layer extended metadata. Level 2 corresponds to a range where the overall suspicion value is greater than or equal to 0.3 and less than 0.6, and the collection dimension set includes basic metadata, TCP layer extended metadata, and NVMe layer extended metadata. Level 3 corresponds to a range where the overall suspicion value is greater than or equal to 0.6 and less than 0.9, and the collection dimension set includes basic metadata, TCP layer, NVMe layer, and block device layer extended metadata. Level 4 corresponds to a range where the overall suspicion value is greater than or equal to 0.9, and the collection dimension set includes all basic metadata and extended metadata from all layers. It should be noted that the specific values ​​of the metadata recording depth level and interval parameters in this application can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the purpose of this invention is achieved. This application does not impose any limitations on these values. Specifically, based on the metadata record depth level, the execution logic of the tracepoint hook for the corresponding I / O stream is updated, adjusting the dimensions of subsequent event data collection. The kernel-mode monitoring module synchronizes the mapped metadata record depth level to the execution context of the tracepoint hook for the corresponding I / O stream. When the tracepoint hook callback function is triggered subsequently, it executes the corresponding collection logic based on the metadata record depth level of the current I / O stream, adjusting the dimensions of event data collection. For I / O streams with lower metadata record depth levels, the callback function only collects basic metadata, executing the fewest instructions possible to minimize execution overhead. For I / O streams with higher metadata record depth levels, the callback function collects extended metadata at the corresponding level to obtain multi-dimensional data supporting anomaly diagnosis. In a specific example, when the metadata record depth level of an I / O stream is updated from level 0 to level 1, the callback function of the corresponding tracepoint hook, when triggered subsequently, will, in addition to writing the basic metadata, additionally read the TCP layer socket parameters, collect the TCP layer extended metadata, and write it to the corresponding fields of the record structure. When the metadata record depth level is updated from level 1 to level 0, the callback function will stop collecting TCP layer extended metadata and only collect basic metadata, restoring to the lowest overhead execution mode.

[0055] Specifically, based on the event data obtained after adjusting the collection dimensions, anomaly detection is completed and anomaly events are generated. The kernel-mode monitoring module, based on the multi-dimensional event data obtained after adjusting the collection dimensions, performs a more comprehensive analysis of I / O flow performance metrics. Anomaly detection is completed through multi-algorithm fusion, determining the specific level and root cause of the anomaly and generating corresponding anomaly events. A comprehensive suspicion value is calculated for each I / O flow based on the collected end-to-end event data, mapping it to the metadata record depth level. The execution logic of the tracepoint hook is updated according to the level, adjusting the collection dimensions. Anomaly detection is completed and anomaly events are generated based on the adjusted event data. Throughout the process, the comprehensive suspicion value quantifies the anomaly risk of each I / O flow, the metadata record depth level is directly bound to the anomaly risk, and the tracepoint hook dynamically adjusts the collection dimensions according to the level. Multi-dimensional extended metadata collection is enabled only for I / O flows with high anomaly risk, while a simplified collection mode is maintained for normal I / O flows. Various technical features work together to obtain all the metadata required for anomaly diagnosis while keeping the execution overhead of the tracepoint hook within the hard constraints of the interrupt context during normal operation.

[0056] In alternative implementations, such as Figure 5 As shown, the calculation of the overall skepticism value for each I / O flow includes: S241: Calculate the latency deviation value of the I / O flow based on I / O flow event data within a fixed statistical period.

[0057] S242: Calculate the trend deterioration value of I / O flow based on I / O flow delay change data within a continuous statistical period.

[0058] S243: Calculate the abnormal event triggering degree of the I / O stream based on the I / O stream error event data within the statistical period.

[0059] S244: Based on the preset weights, the latency deviation value, the trend deterioration value, and the abnormal event trigger value are combined to obtain the comprehensive suspicion value of the I / O flow.

[0060] In this embodiment of the application, the calculation of the comprehensive skepticism value for each I / O flow includes four specific execution steps. The four steps are based on the aggregated I / O flow performance index data to complete the calculation of the comprehensive skepticism value, and the output of the four steps is the final comprehensive skepticism value.

[0061] Specifically, based on I / O stream event data within a fixed statistical period, the latency deviation of the I / O stream is calculated. The kernel-mode monitoring module first calculates the measured average latency of the I / O stream within the current statistical period based on the event data collected. The measured average latency is obtained by calculating the arithmetic mean of the end-to-end latency of all I / O requests for that I / O stream within the current statistical period. The end-to-end latency is calculated by the difference between the block-layer delivery timestamp and the block-layer completion timestamp of the same I / O request. The kernel-mode monitoring module calculates the latency deviation of the I / O stream within the current statistical period based on the measured average latency and the dynamic latency baseline of the I / O stream. Finally, the trend deterioration of the I / O stream is calculated based on I / O stream latency change data within continuous statistical periods. The kernel-mode monitoring module acquires the measured average latency data of the I / O stream over multiple consecutive statistical periods, calculates the cumulative latency increase within each period, and calculates the trend deterioration value of the I / O stream based on the cumulative latency increase and the dynamic latency baseline. The trend deterioration value is fixed between 0 and 1. The module also calculates the abnormal event triggering value of the I / O stream based on I / O stream error event data within the statistical periods. The kernel-mode monitoring module counts the number of I / O errors, timeouts, PDU parsing failures, and other abnormal events occurring in the I / O stream within the current statistical period, as well as the total number of I / O requests for the I / O stream within the current statistical period. Based on the ratio of the number of abnormal events to the total number of I / O requests, the module calculates the abnormal event triggering value of the I / O stream, which is also fixed between 0 and 1. Finally, the module integrates the latency deviation value, trend deterioration value, and abnormal event triggering value using preset weights to obtain the overall skepticism value of the I / O stream. The kernel-mode monitoring module multiplies the values ​​of the three dimensions by their respective preset weights, and then adds the three weighted values ​​together to obtain the final comprehensive skepticism score.

[0062] In this embodiment, the dynamic baseline of I / O flow latency is updated based on the exponential moving average algorithm. The exponential moving average algorithm is a smoothing algorithm for time series data. It assigns higher weights to recent observations and decreasing weights to historical observations, adaptively tracking trend changes in time series data while smoothing temporary random fluctuations, avoiding misjudging normal business fluctuations as anomalies. The update formula for the dynamic baseline latency is:

[0063] In this embodiment of the application, the meanings of each letter in the formula are as follows: This represents the dynamic baseline of I / O flow delay after the t-th statistical period update, in microseconds. represents the historical dynamic baseline of I / O flow latency in the (t-1)th statistical period, in microseconds. Dt represents the measured average latency of the I / O flow within the t-th statistical period, in microseconds. k represents the smoothing coefficient of the exponential moving average algorithm. It should be noted that the value of the smoothing coefficient k determines the response speed of the dynamic baseline to recent measured latency. A larger value of k results in a more sensitive dynamic baseline to changes in recent measured latency, and a faster tracking speed. A smaller value of k results in a stronger smoothing effect of the dynamic baseline and a better filtering effect on temporary fluctuations.

[0064] Specifically, the relative delay deviation rate is calculated based on the difference between the measured average delay of the I / O flow within the statistical period and the dynamic baseline of the delay. The formula for calculating the relative delay deviation rate is:

[0065] In the formula, Let represent the relative delay deviation rate of the I / O flow within the t-th statistical period, and Dt represent the measured average delay within the t-th statistical period. This represents the dynamic baseline of delay after the update in the t-th statistical period. The relative delay deviation rate is used to quantify the degree of deviation of the measured average delay from the dynamic baseline. The larger the value, the greater the deviation of the delay from the normal baseline. When the measured average delay is less than or equal to the dynamic baseline, the relative delay deviation rate is 0, representing no delay deviation.

[0066] Specifically, based on the relative latency deviation rate and a preset deviation threshold, a latency deviation value for the I / O stream is generated. The formula for calculating the latency deviation value is:

[0067] In the formula, Sdev,t represents the latency deviation value of the I / O flow in the t-th statistical period, Rdev,t represents the relative latency deviation rate in the t-th statistical period, and α represents the preset deviation threshold. The max and min functions are used to strictly limit the latency deviation value to between 0 and 1. When the relative latency deviation rate is greater than or equal to the preset deviation threshold, the latency deviation value is 1. When the relative latency deviation rate is less than or equal to 0, the latency deviation value is 0. The preset deviation threshold α represents the minimum relative latency deviation rate that triggers the maximum deviation value. The default value of 0.2 means that when the measured average latency exceeds the dynamic baseline by 20%, the latency deviation value reaches the maximum value of 1. The preset deviation threshold can be adjusted through monitoring configuration parameters. Different values ​​can be set according to the latency sensitivity of different business scenarios. The higher the latency sensitivity of the business scenario, the smaller the value of the preset deviation threshold. It should be noted that the specific value of the preset deviation threshold in this application can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the purpose of this application is achieved. This application does not limit this.

[0068] In this embodiment of the application, the calculation of the trend deterioration value of the I / O flow includes three specific execution steps. These three steps are based on the delay data of continuous statistical periods to complete the quantitative calculation of the trend deterioration.

[0069] Specifically, the measured average latency sequence of the I / O flow within a continuous statistical period is obtained. The kernel-mode monitoring module obtains the measured average latency data of the I / O flow for the current statistical period and the previous N consecutive statistical periods, forming a latency sequence of length N+1.

[0070] Specifically, the cumulative increase in latency over consecutive statistical periods is calculated. The kernel-mode monitoring module iterates through the latency sequence, calculates the latency difference between two adjacent statistical periods, and only accumulates the positive latency differences to obtain the cumulative increase in latency over consecutive statistical periods. The formula for calculating the cumulative increase in latency is:

[0071] In the formula, ΔDt represents the cumulative increase in delay calculated in the t-th statistical period, in microseconds. Di represents the measured average delay in the i-th statistical period. N represents the number of consecutive statistical periods.

[0072] Specifically, based on the cumulative increase in latency and the dynamic baseline of latency, a trend deterioration value for the I / O flow is generated. The formula for calculating the trend deterioration value is:

[0073] In the formula, This represents the degree of deterioration in the trend of I / O flow during the t-th statistical period. This indicates the cumulative increase due to delay. This represents the delayed dynamic baseline for the t-th statistical period. This represents a preset trend threshold. The `max` and `min` functions are used to strictly limit the value of the trend deterioration degree to between 0 and 1. When the cumulative increase in latency is greater than or equal to the dynamic latency baseline multiplied by the preset trend threshold, the trend deterioration degree value is 1. When the cumulative increase in latency is equal to 0, the trend deterioration degree value is 0. The preset trend threshold β represents the minimum cumulative latency increase that triggers the maximum trend deterioration degree value. The default value of 0.3 means that when the cumulative latency increase over three consecutive statistical periods reaches 30% of the dynamic baseline, the trend deterioration degree value reaches its maximum value of 1. The preset trend threshold can be adjusted through monitoring configuration parameters to adapt to the sensitivity of different business scenarios to latency degradation trends. It should be noted that the specific value of the preset trend threshold β in this application can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the purpose of this application is achieved. This application does not limit this setting.

[0074] In alternative implementations, such as Figure 6 As shown, the calculated delay deviation values ​​for the I / O flow include: S281: Update the dynamic baseline of I / O flow latency based on the exponential moving average algorithm.

[0075] S282: Calculate the relative delay deviation rate based on the difference between the measured average delay of I / O flow and the dynamic delay baseline within the statistical period.

[0076] S283: Generate the delay deviation value of the I / O stream based on the relative delay deviation rate and the preset deviation threshold.

[0077] In this embodiment of the application, the calculation of the abnormal event triggering degree of the I / O stream includes two specific execution steps. The two steps are based on the error event data within the statistical period to complete the quantitative calculation of the abnormal event triggering degree.

[0078] Specifically, the kernel-mode monitoring module counts the number of abnormal events and the total number of I / O requests within the current statistical period. It iterates through the event data collected within the current statistical period, counting the total number of abnormal events occurring in the I / O stream, such as I / O errors, timeouts, PDU parsing failures, and queue full events. Simultaneously, it counts the total number of I / O requests completed by the I / O stream within the current statistical period.

[0079] Specifically, an exception triggering score is generated for the I / O stream based on the ratio of the number of exception events to the total number of I / O requests. The formula for calculating the exception triggering score is:

[0080] In the formula, This represents the abnormal event triggering degree of the I / O flow within the t-th statistical period. This represents the total number of abnormal events within the t-th statistical period. γ represents the total number of I / O requests within the t-th statistical period. γ is a preset error rate threshold. ε is a minimum constant used to avoid division-by-zero errors. The max and min functions are used to strictly limit the value of the abnormal event triggering degree to between 0 and 1. When the ratio of the number of abnormal events to the total number of I / O requests is greater than or equal to the preset error rate threshold, the abnormal event triggering degree is 1. When the number of abnormal events is 0, the abnormal event triggering degree is 0. The preset error rate threshold γ represents the minimum error rate required to trigger the maximum abnormal event triggering degree. The default value of 0.001 means that when the error rate reaches 0.1%, the abnormal event triggering degree reaches its maximum value of 1. The preset error rate threshold can be adjusted through monitoring configuration parameters to adapt to the tolerance of different business scenarios for abnormal events. It should be noted that the specific value of the preset error rate threshold γ in this application can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the purpose of this application is achieved. This application does not limit this.

[0081] In alternative implementations, such as Figure 7 As shown, the step of updating the tracepoint hook execution logic of the corresponding I / O stream according to the metadata record depth level and adjusting the collection dimensions of subsequent event data includes: S261: Map the overall skepticism value to discrete metadata record depth levels, with each metadata record depth level corresponding to a fixed set of event data collection dimensions.

[0082] S262: Synchronize the metadata record depth level to the tracepoint hook execution context of the corresponding I / O stream using atomic variables.

[0083] S263: When the tracepoint hook is triggered, read the metadata record depth level stored in the atomic variable and execute the event data write operation in the corresponding collection dimension set.

[0084] In this embodiment, the overall skepticism value is mapped to discrete metadata record depth levels, with each metadata record depth level corresponding to a fixed set of event data collection dimensions. A predefined mapping table between numerical ranges and levels is used in the kernel-mode monitoring module. Based on the range of the overall skepticism value of the I / O flow, the corresponding metadata record depth level is retrieved. Each metadata record depth level corresponds to a fixed collection dimension bitmap. Each bit in the bitmap corresponds to an extended metadata dimension. When the corresponding bit is 1, it indicates that the extended metadata for that dimension needs to be collected. When the corresponding bit is 0, it indicates that the extended metadata for that dimension does not need to be collected. The collection dimension bitmap is stored using unsigned integers, allowing for quick determination of whether the corresponding dimension's extended metadata needs to be collected through bitwise operations. This results in extremely high execution efficiency and adapts to the execution requirements of interrupt contexts.

[0085] Specifically, the metadata record depth level is synchronized to the corresponding I / O stream's tracepoint hook execution context via atomic variables. The kernel-mode monitoring module creates a corresponding entry in the watchlist hash table for each I / O stream. Each entry contains an atomic variable of type `atomic_t` to store the metadata record depth level of that I / O stream. When the kernel-mode monitoring module updates the metadata record depth level of an I / O stream, it updates the value of the atomic variable using the `atomic_set` atomic operation. Atomic operations are single-CPU instruction-level operations, eliminating multi-core concurrency contention and preventing dirty data. The watchlist hash table is protected by RCU (Remote Control Unit). Read operations are executed in interrupt context, without locks or blocking. Write operations are executed in process context, and RCU ensures memory access safety, preventing interrupt context access to already released memory. When a tracepoint hook is triggered, the metadata record depth level stored in the atomic variable is read, and the event data write operation within the corresponding collection dimension set is executed. When the callback function of the tracepoint hook is triggered, it first queries the watchlist hash table within the RCU read critical section using the unique identifier of the I / O stream. If no corresponding entry is found, it means that the metadata record depth level of that I / O stream is 0, and only basic metadata is collected. If a corresponding entry is found, it reads the metadata record depth level stored in the entry using the atomic_read operation, and performs the corresponding dimension event data write operation according to the collection dimension bitmap corresponding to the level.

[0086] The RCU mechanism used in this embodiment is short for Read-Copy-Update, an advanced synchronization mechanism in the Linux kernel used to implement lock-free concurrent access. The principle of RCU is that read operations access shared data directly without any locks. For write operations, a copy of the shared data is first created, modifications are made to the copy, and then, after all existing read operations have completed, the original data pointer is replaced with the modified copy pointer. Finally, the memory of the original data is released. The RCU mechanism perfectly suits the scenario in this embodiment; read operations in interrupt context are lock-free and non-blocking, without affecting execution efficiency. Write operations in process context do not interfere with the execution of read operations, ensuring the safety of concurrent access to the watchlist hash table.

[0087] In alternative implementations, such as Figure 8 As shown, the execution of the event data writing operation within the corresponding collection dimension set includes: S264: Match the collection dimension set corresponding to the metadata record depth level and write the basic metadata to the pre-allocated record structure.

[0088] S265: Determine whether the collection dimension set contains extended metadata dimensions. If it does, read the event parameters of the corresponding kernel subsystem and write the extended metadata to the pre-allocated record structure.

[0089] S266: After completing the writing of event data for all matching dimensions, write the record structure to the per-CPU circular buffer.

[0090] Specifically, the collection dimension set corresponding to the metadata record depth level is matched, and the basic metadata is written to the pre-allocated record structure. The callback function first obtains a pre-allocated record structure instance from the per-CPU memory pool, and writes basic metadata such as timestamp, event type, I / O stream unique identifier, and command identifier into the basic fields of the record structure.

[0091] Specifically, the system determines whether the collection dimension set contains extended metadata dimensions. If so, it reads the event parameters of the corresponding kernel subsystem and writes the extended metadata to the pre-allocated record structure. The callback function uses bitwise operations to determine whether to collect the corresponding extended metadata based on the collection dimension bitmap corresponding to the metadata record depth level. If the corresponding bit in the bitmap is 1, it reads the event parameters of the corresponding kernel subsystem and writes them to the corresponding extended field of the record structure. If the corresponding bit is 0, it skips the collection operation for that dimension and does not execute any related instructions. This bitwise operation enables rapid determination of the collection dimension without branching, jumping, or looping, resulting in extremely high execution efficiency and allowing for strict control over the number of instructions executed in a single callback function. In a specific example, if the metadata record depth level is 1 and the bit corresponding to the TCP layer extended metadata in the collection dimension bitmap is 1, the callback function will read parameters such as the smooth round-trip time, congestion window, retransmission count, and connection status from the TCP socket structure and write them to the TCP layer extended field of the record structure. If the metadata record depth level is 0 and all bits corresponding to the extended metadata in the collection dimension bitmap are 0, the callback function will skip all extended metadata collection operations, write only the basic metadata, and then directly complete the execution. After writing event data for all matching dimensions, the record structure is written to the per-CPU circular buffer. The callback function, after completing the writing of event data for all matching dimensions, writes the filled record structure to the per-CPU circular buffer corresponding to the current CPU, updates the producer pointer, and simultaneously updates the global statistics counter, completing the entire data collection process.

[0092] In alternative implementations, such as Figure 9 As shown, after completing anomaly detection and generating anomaly events based on the event data obtained after adjusting the collection dimensions, the process further includes: S271: Based on the event data obtained after adjusting the collection dimensions, complete the anomaly root cause location and update the comprehensive suspicion value of the corresponding I / O stream.

[0093] S272: If the overall skepticism value of the I / O flow is lower than the preset threshold within a continuous statistical period, the metadata record depth level of the corresponding I / O flow will be reset to the initial level, and the collection of extended metadata dimensions will be stopped.

[0094] Specifically, based on the event data obtained after adjusting the collection dimensions, the root cause of the anomaly is located, and the overall suspicion value of the corresponding I / O flow is updated. The kernel-mode monitoring module, based on multi-dimensional extended metadata, completes the root cause location of the anomaly, determines the specific level and cause of the anomaly, and recalculates the overall suspicion value of the I / O flow based on more comprehensive performance indicator data, updating the corresponding metadata record depth level. If the root cause location confirms that the anomaly risk continues to increase, the metadata record depth level is increased, and more dimensions of metadata collection are enabled. If the root cause location confirms that the anomaly is intermittent and without a continuous deterioration trend, the metadata record depth level is decreased, and the collection of extended metadata for the corresponding dimension is disabled. If the overall suspicion value of the I / O flow is lower than a preset threshold within a continuous statistical period, the metadata record depth level of the corresponding I / O flow is reset to the initial level, and the collection of extended metadata dimensions is stopped. The default preset threshold is 0.1. If the overall suspicion value of an I / O flow is below 0.1 for three consecutive statistical periods, it indicates that the abnormal risk of the I / O flow has been eliminated. The kernel-mode monitoring module resets the metadata recording depth level of the I / O flow to level 0 and deletes the corresponding entry from the watchlist hash table. When the tracepoint hook is triggered subsequently, only basic metadata is collected, restoring the execution mode to the lowest overhead. The specific values ​​of the preset threshold, statistical period, and overall suspicion value can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the purpose of this invention is achieved. This application does not limit these settings. In a specific example, taking the NVMe / TCP protocol performance monitoring scenario of a distributed storage cluster of a certain operator as an example, this cluster carries database services, with 1200 active I / O streams in a single cluster, and deploys the monitoring system provided in this application embodiment. The user starts the user-space control module via command line, passing in the configuration file path. The configuration file sets the statistical period to 100 milliseconds, the smoothing coefficient k to 0.125, the preset deviation threshold α to 0.2, the preset trend threshold β to 0.3, the preset error rate threshold γ to 0.001, and the weights to latency deviation 0.4, trend deterioration 0.35, and abnormal event triggering 0.25. The user-space control module parses the configuration file to generate monitoring configuration parameters and sends them to the kernel-space monitoring module via an ioctl system call. After receiving the configuration parameters, the kernel-space monitoring module completes initialization operations such as per-CPU ring buffer allocation, tracepoint hook registration, and kernel work queue creation, and starts the monitoring process. All 1200 I / O streams in the cluster are operating normally. The measured average latency of a single I / O stream is stable at around 200 microseconds, with no latency degradation trend and no abnormal events occurring. The kernel-mode monitoring module calculates a comprehensive suspicion value for each I / O stream in each statistical period. The comprehensive suspicion values ​​for all I / O streams are below 0.1, the metadata record depth level is 0, and they are not included in the observation list. When the tracepoint hook is triggered, only basic metadata collection is performed, with 42 instructions executed in a single callback function. The overall CPU overhead is stable at 0.09%, having no impact on business I / O operations. One I / O stream, uniquely identified as controller_id=0x0010 and qid=0x0003, corresponding to a database volume, exhibits progressive latency degradation. In the t-th statistical period, the measured average latency of this stream rises to 220 microseconds, with a relative latency deviation rate of 0.1 and a latency deviation value of 0.5. The latency increased for two consecutive cycles, with a cumulative increase of 20 microseconds and a trend deterioration index of 0.33. No abnormal events occurred, and the abnormal event trigger index was 0. The overall suspicion index was 0.4×0.5+0.35×0.33+0.25×0=0.3155, falling within the 0.3-0.6 range, and the metadata recording depth level was updated to level 2. The kernel-mode monitoring module added this flow to the watchlist and synchronized the level to the tracepoint hook execution context through atomic operations. When the tracepoint hook was subsequently triggered, it read the level 2 metadata recording depth. In addition to the basic metadata, it also collected extended metadata from the TCP and NVMe layers. The number of instructions executed in a single callback function was 225, still strictly controlled within the hard constraints of the interrupt context. In the following two statistical cycles, the measured average latency of this flow continued to rise to 280 microseconds, the overall suspicion index rose to 0.75, the metadata recording depth level was updated to level 3, and the collection of extended metadata from the block device layer was enabled.Based on the collected multi-dimensional extended metadata, the kernel-mode monitoring module discovered that the number of pending commands in the NVMe commit queue of this flow consistently exceeded 80% of the maximum queue depth, indicating persistent congestion in the block device layer I / O scheduling queue. The root cause was ultimately identified as improper configuration of the block device layer I / O scheduler parameters, leading to queue congestion and increased latency. Based on the diagnostic results, the operations team adjusted the I / O scheduler parameters and queue depth for this flow. The measured average latency of this flow immediately dropped to 210 microseconds, returning to the normal baseline range. For the next three consecutive statistical periods, the overall suspicion value of this flow remained below 0.1. The kernel-mode monitoring module removed this flow from the watchlist, reset the metadata record depth level to level 0, and restored the tracepoint hook to a minimalist mode that only collects basic metadata. The overall system CPU overhead dropped to a normal level of 0.09%. Throughout the entire anomaly period, there was no service interruption. The monitoring system fully collected diagnostic data for the entire anomaly period, located the root cause of the anomaly, and reduced the performance impact on the service to a negligible level. It should be noted that the specific values ​​of the parameters or thresholds mentioned above in this application are merely illustrative. In practical applications, those skilled in the art can flexibly set them according to actual needs, as long as the purpose of this application can be achieved. This application does not limit them in this regard.

[0095] In summary, in the field of performance monitoring of the NVMe / TCP protocol stack (a non-volatile memory host controller interface specification based on the Transmission Control Protocol), traditional solutions rely on user-space agents to read aggregate statistics. These solutions only show latency values ​​but cannot pinpoint whether the latency occurs at the block layer, driver layer, protocol layer, or target end. Alternatively, kernel probes indiscriminately record all I / O flows, causing the accumulated execution time of the probe function in the intermediate context to consume processor resources allocated to business I / O, thus compromising the low-latency characteristics of the storage system. This application uses a comprehensive skepticism value as a feedback signal and metadata recording depth level as a control variable to drive real-time adjustment of the tracepoint probe's collection granularity. Under normal conditions, the vast majority of healthy I / O flows consume very few monitoring resources. When an I / O flow exhibits abnormal signs such as latency deviation, deteriorating trends, or increased errors, monitoring resources are automatically allocated to that flow, dynamically enabling the collection of multi-dimensional extended metadata.

[0096] This application utilizes atomic variables and RCU to protect the hash table. The update end completes the level setting through a single atomic write instruction, and the probe end obtains it immediately through a single atomic read instruction, without any lock operations. The probe needs to jump to execute different acquisition logic with a minimal number of instructions according to the level instructions. This application maps each level to a acquisition dimension bitmap. The probe determines the on / off state of each dimension through bit test instructions, controlling the fluctuation of single execution time to the nanosecond level. At the moment of dynamic adjustment of acquisition dimensions, the write path of basic metadata remains constant. The pre-allocated per-CPU circular buffer and fixed record structure ensure that there is no dynamic memory allocation in the interrupt context, and the basic timing information such as timestamps and event types throughout the entire lifecycle of the I / O stream remains continuous and consistent. The comprehensive suspicion value establishes a dynamic latency baseline for each I / O stream using an exponential moving average, and generates a quantified risk value by fusing latency deviation, trend deterioration, and abnormal event triggering degree, which is mapped to discrete metadata record depth levels. When the probe is triggered, it reads the bitmap corresponding to the level, acquires the required extended metadata bit by bit, and writes it to the circular buffer exclusively used by the current CPU. The kernel work queue periodically consumes event data, reassesses the risk of each flow and updates its level. I / O flows with continuously declining scores are removed from the watchlist, and the collection logic reverts to basic mode.

[0097] The kernel work queue used in this embodiment is a mechanism in the Linux kernel used to defer work to process context execution. Its principle is to create kernel threads in the kernel, add the work items to be executed to the work queue, and then schedule the execution of the kernel threads in process context. The execution environment of the kernel work queue is process context, without the execution constraints of interrupt context, allowing for complex computational operations, memory allocation operations, and blocking operations without affecting the handling of service interruptions. In this embodiment, all complex computational operations such as comprehensive suspicion value calculation, anomaly detection, and root cause localization are executed in the kernel work queue, without interfering with the interrupt context execution of tracepoint hooks, thus ensuring low-latency characteristics of service I / O.

[0098] The circular buffer used in this embodiment is a fixed-size circular data buffer, employing a producer-consumer model for data reading and writing. The circular buffer's memory is pre-allocated contiguous physical memory. Read and write operations are implemented by moving producer and consumer pointers. When a pointer reaches the end of the buffer, it automatically loops back to the beginning, forming a circular structure. This embodiment uses a per-CPU circular buffer, with each CPU corresponding to an independent circular buffer. The producer is the tracepoint hook callback function in the current CPU interrupt context, and the consumer is the kernel work queue in the process context. This single-producer, single-consumer model enables lock-free read and write operations, ensuring data read and write security without the need for locking. It boasts extremely high execution efficiency and is perfectly suited to the high-concurrency data acquisition scenarios described in this embodiment.

[0099] The tracepoint hook used in this embodiment is a static probe mechanism natively provided by the Linux kernel. Its implementation principle is that tracepoint hooks are predefined in the code paths within the kernel source code, and probe jump instructions are reserved in the kernel code during the compilation phase. During kernel startup, the jump instructions corresponding to tracepoint hooks without registered callback functions are replaced with nop operations, having no impact on the execution of the native kernel code paths. When a kernel module registers a tracepoint hook's callback function, the kernel modifies the reserved jump instructions to jump to the callback function. When the kernel executes the corresponding code path, the callback function is triggered. Tracepoint hooks are characterized by high stability, low execution overhead, and non-intrusiveness, forming the basis for implementing end-to-end event collection in this embodiment.

[0100] The loadable kernel module used in this embodiment is a mechanism provided by the Linux kernel that allows developers to compile kernel function code into independent modules, which are dynamically loaded into the kernel space for execution during system runtime without recompiling the kernel or restarting the system. The loadable kernel module runs in kernel space, can directly access all kernel functions and data structures, can implement all kernel-mode functionalities, and possesses the flexibility of dynamic loading and unloading. It serves as the implementation carrier for the kernel-mode monitoring module in this embodiment.

[0101] The character device used in this embodiment is one of the three basic device types in the Linux kernel. Character devices read and write data in a byte stream manner and do not support random access. Character devices are uniquely identified by a major device number and a minor device number. The major device number identifies the corresponding driver, and the minor device number identifies the specific device instance managed by the driver. Character devices expose file operation interfaces to user space through the `file_operations` structure. User-space applications can communicate with the kernel-mode driver through standard file operating system calls such as `open`, `ioctl`, `read`, and `write`. In this embodiment, the kernel-mode monitoring module uses the character device to implement the exchange of configuration parameters and control commands between the user-mode control module and the kernel-mode monitoring module.

[0102] The proc file system used in this embodiment is a virtual file system provided by the Linux kernel. The proc file system does not exist on actual physical storage devices; its content is dynamically generated by the kernel at runtime. It is used to expose the kernel's runtime status information to user space and supports user space passing configuration parameters to the kernel by reading and writing proc file nodes. The proc file system's file operation interface is simple to implement and easy to access, making it a common method for communication between user space and kernel space in the Linux kernel. In this embodiment, the kernel-mode monitoring module exports statistical data and abnormal events to the user-mode control module through the proc file system, achieving efficient transmission of monitoring data.

[0103] The NVMe / TCP protocol involved in this application embodiment, short for Non-Volatile Memory Expressover TCP, is an NVMe storage protocol based on TCP / IP networks. It allows the host to access remote NVMe storage devices via standard Ethernet without requiring dedicated Fibre Channel or InfiniBand networks. It features flexible deployment, low cost, and good compatibility, and is a widely used storage network protocol in current distributed storage systems. The complete I / O processing flow of the NVMe / TCP protocol includes: the host-side block device layer generating an I / O request; the NVMe driver layer encapsulating the I / O request into an NVMe protocol command; the NVMe / TCP protocol layer encapsulating the NVMe command into a PDU and handing it over to the TCP network layer for transmission; the target-side TCP network layer receiving the data packet; the NVMe / TCP protocol layer parsing the NVMe command; the target-side NVMe driver layer sending the command to the backend storage device; and after the storage device completes the I / O request, returning the response to the host through the reverse path. The tracepoint hook in this application embodiment covers all code paths in the entire NVMe / TCP protocol I / O processing flow, enabling end-to-end full-link event collection.

[0104] This application also provides an NVMe / TCP protocol end-to-end performance data acquisition and processing system, such as... Figure 10 As shown, the system includes: User-mode control module 11 parses the configuration file to generate monitoring configuration parameters, and sends the monitoring configuration parameters to the kernel-mode monitoring module through the character device interface to complete the initialization of the monitoring environment.

[0105] The kernel-mode monitoring module 12 registers the kernel tracepoint hooks corresponding to the entire NVMe / TCP protocol link. It collects the full-link event data of the corresponding I / O stream through the tracepoint hooks, writes the collected event data into a pre-allocated per-CPU ring buffer, performs anomaly detection based on the event data, and generates anomaly events.

[0106] The monitoring report module 13, the kernel-mode monitoring module, exports statistical data and abnormal events to the user-mode control module through the virtual file system, and the user-mode control module performs abnormal root cause location.

[0107] The technical effects of the NVMe / TCP protocol end-link performance data acquisition and processing system provided in this application are the same as the corresponding methods, and will not be elaborated here.

[0108] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0109] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for collecting and processing full-link performance data of an NVMe / TCP protocol, characterized in that, The method comprises: The user state control module parses the configuration file to generate monitoring configuration parameters, and sends the monitoring configuration parameters to the kernel state monitoring module through a character device interface to complete monitoring environment initialization; The kernel state monitoring module registers a kernel tracepoint hook corresponding to a full link of an NVMe / TCP protocol, collects full link event data of a corresponding I / O stream through the tracepoint hook, writes the collected event data into a pre-allocated per-CPU ring buffer, completes abnormality detection based on the event data, and generates an abnormal event; The kernel state monitoring module exports the statistical data and the abnormal event to the user state control module through a virtual file system, and the user state control module performs abnormal root cause positioning.

2. The NVMe / TCP protocol full link performance data collection and processing method according to claim 1, characterized in that, The collection of the full link event data of the corresponding I / O stream through the tracepoint hook comprises: The block device layer tracepoint hook is used to collect block layer event data of an I / O request, and the NVMe drive layer tracepoint hook is used to collect NVMe command event data of the I / O request; The NVMe / TCP protocol layer tracepoint hook is used to collect TCP transmission event data of the I / O request; The Target end tracepoint hook is used to collect back-end processing event data of the I / O request.

3. The NVMe / TCP protocol full link performance data collection and processing method according to claim 2, characterized in that, The collection of the block layer event data of the I / O request through the block device layer tracepoint hook comprises: When the tracepoint hook is triggered, a fast path check is performed, and an I / O request corresponding to the monitoring configuration parameters is matched; A predefined record structure is allocated for the matched I / O request, and a timestamp and an event type of event triggering are recorded; The record structure is written into a per-CPU ring buffer of a corresponding CPU, and a statistical counter is updated.

4. The NVMe / TCP protocol full link performance data collection and processing method according to claim 1, characterized in that, The completion of the abnormality detection based on the event data and the generation of the abnormal event comprise: Based on the collected full link event data, a comprehensive suspicion degree value of each I / O stream is calculated, and the comprehensive suspicion degree value is used to quantify an abnormal risk level of the I / O stream; According to the comprehensive suspicion degree value, a metadata record depth level of the corresponding I / O stream is obtained; According to the metadata record depth level, tracepoint hook execution logic of the corresponding I / O stream is updated, and a collection dimension of subsequent event data is adjusted; Based on the event data obtained after the collection dimension is adjusted, the abnormality detection is completed, and an abnormal event is generated.

5. The NVMe / TCP protocol full link performance data collection and processing method according to claim 4, characterized in that, The calculation of the comprehensive suspicion degree value of each I / O stream comprises: Based on I / O stream event data in a fixed statistical period, a delay deviation degree value of the I / O stream is calculated; Based on I / O stream delay change data in a continuous statistical period, a trend deterioration degree value of the I / O stream is calculated; Based on I / O stream error event data in a statistical period, an abnormal event triggering degree value of the I / O stream is calculated; The delay deviation degree value, the trend deterioration degree value, and the abnormal event triggering degree value are fused based on a preset weight to obtain the comprehensive suspicion degree value of the I / O stream.

6. The NVMe / TCP protocol full link performance data collection and processing method according to claim 5, characterized in that, The calculation of the delay deviation degree value of the I / O stream comprises: The delay dynamic baseline of the I / O stream is updated based on an exponential moving average algorithm; The relative delay deviation rate is calculated based on the difference between the measured average delay of the I / O flow and the dynamic delay baseline within the statistical period; the delay deviation value of the I / O flow is generated based on the relative delay deviation rate and the preset deviation threshold.

7. The NVMe / TCP protocol full link performance data collection and processing method according to claim 4, characterized in that, The step of updating the tracepoint hook execution logic of the corresponding I / O stream based on the metadata record depth level and adjusting the collection dimensions of subsequent event data includes: The overall skepticism value is mapped to discrete metadata record depth levels, and each metadata record depth level corresponds to a fixed set of event data collection dimensions; Metadata record depth levels are synchronized to the corresponding I / O stream's tracepoint hook execution context using atomic variables; When the tracepoint hook is triggered, the metadata record depth level stored in the atomic variable is read, and the event data writing operation in the corresponding collection dimension set is executed.

8. The NVMe / TCP protocol full link performance data collection and processing method according to claim 7, characterized in that, The execution of the event data writing operation within the corresponding collection dimension set includes: Match the collection dimension set corresponding to the depth level of the metadata record and write the basic metadata to the pre-allocated record structure; Determine whether the collection dimension set contains extended metadata dimensions. If it does, read the event parameters of the corresponding kernel subsystem and write the extended metadata to the pre-allocated record structure. After writing event data for all matching dimensions, the record structure is written to the per-CPU circular buffer.

9. The NVMe / TCP protocol full link performance data collection and processing method according to claim 4, characterized in that, After completing anomaly detection and generating anomaly events based on the event data obtained after adjusting the collection dimensions, the process also includes: Based on the event data obtained after adjusting the collection dimensions, the root cause of the anomaly is located and the comprehensive suspicion value of the corresponding I / O stream is updated. If the overall skepticism value of the I / O flow is lower than the preset threshold within a continuous statistical period, the metadata record depth level of the corresponding I / O flow will be reset to the initial level, and the collection of extended metadata dimensions will be stopped.

10. An NVMe / TCP protocol full link performance data collection and processing system, characterized in that, The system includes: The user-mode control module parses the configuration file to generate monitoring configuration parameters, and then sends these parameters to the kernel-mode monitoring module via the character device interface to complete the initialization of the monitoring environment. The kernel-mode monitoring module registers kernel tracepoint hooks corresponding to the entire NVMe / TCP protocol link. It collects the entire link event data of the corresponding I / O stream through the tracepoint hooks, writes the collected event data into a pre-allocated per-CPU circular buffer, performs anomaly detection based on the event data, and generates abnormal events. The monitoring and reporting module, which is part of the kernel-mode monitoring module, exports statistical data and abnormal events to the user-mode control module via the virtual file system. The user-mode control module then locates the root cause of the abnormalities.

Images