Progressive memory latency monitoring and diagnostic method for virtual memory subsystems

Abstract

The present application relates to a progressive memory delay monitoring and diagnosis method for a virtual memory subsystem, comprising: mounting a BPF program to a plurality of key event points on a kernel; when the key event points are triggered, obtaining state information of the corresponding events; encapsulating the state information as events and writing into a perf ring buffer associated with an output_map; a user state program asynchronously reads the encapsulated events from the perf ring buffer, classifies and aggregates according to the process and the event type, generates a delay terminal log and a histogram, and outputs a diagnosis result. The present application combines the state retention capability of eBPF, the kernel context access capability and the event model of the virtual memory subsystem through a systematic architecture design, and constructs a closed-loop monitoring system for a diagnosis target. High-precision and high-reliability memory operation delay measurement is realized.

Description

Technical Field

[0001] This invention relates to the field of memory fault diagnosis technology, and specifically to a progressive memory latency monitoring and diagnosis method for virtual memory subsystems. Background Technology

[0002] In modern server environments, efficient memory resource management directly impacts system performance and service quality. When system memory pressure increases, the Linux kernel triggers different levels of memory reclamation mechanisms according to priority, forming a progressive response path from light to heavy and escalating step by step: kswapd reclamation: a background asynchronous thread periodically reclaims pages to avoid process blocking; Direct Reclaim: the current process performs reclamation synchronously when memory allocation is insufficient, causing significant delays; Memory Compaction: triggered when allocating large pages fails to solve memory fragmentation problems, but execution is time-consuming and can easily cause stuttering.

[0003] Traditional performance analysis tools (such as perf and ftrace) can capture the above events through kprobe or tracepoint, but their monitoring methods are mostly stateless event stream records, lacking the ability to correlate event context, making it difficult to accurately calculate the latency of the complete behavior (such as the duration from mm_vmscan_direct_reclaim_begin to mm_vmscan_direct_reclaim_end), and even more difficult to construct the causal chain of "slow kswapd response to frequent direct reclaim to compaction trigger".

[0004] CN119046021B discloses a Linux system memory reclamation control method and system based on eBPF. It uses an eBPF program to attach to critical paths in the kernel (such as shrink_folio_list) and handles these events in user-space programs to monitor memory reclamation behavior. However, it has the following drawbacks:

[0005] 1. Lack of a unified latency monitoring mechanism: Current system tools (such as vmstat, perf, ftrace) can only provide discrete event logs or statistical summaries, and cannot perform end-to-end latency quantification for critical paths such as direct reclaim, kswapd, and compaction, establish chain analysis capabilities from kswapd to direct reclaim and then to compaction, and cannot achieve cross-path correlation analysis.

[0006] 2. Coarse monitoring granularity: BPF Map is mainly used to pass filtering configurations and is not designed for aggregating monitoring data. It does not support statistical analysis of multiple event chains.

[0007] 3. State loss, inability to accurately match begin / end events: Traditional methods rely on user-space tools to parse ftrace logs, which makes it difficult to guarantee the accurate pairing of begin and end events. The complete latency from begin to end is not calculated, and the complete end-to-end latency distribution cannot be obtained. This is especially prone to misalignment or loss in high-concurrency, multi-process scenarios.

[0008] 4. Lack of diagnostic-oriented data aggregation capabilities: Existing tools output raw logs or simple counts, lacking diagnostic indicators such as latency distribution and success rate by process, cgroup, node (NUMA node), etc., making it difficult to locate the root cause. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention provides a progressive memory latency monitoring and diagnosis method for virtual memory subsystems, comprising:

[0010] Step S1: Load the BPF program into the kernel subsystem and attach the BPF program to multiple critical event points on the kernel;

[0011] Step S2: Initialize the hash table - state map - for recording key event states of users;

[0012] Step S3: When a critical event is triggered, obtain the status information of the corresponding critical event and write it into the statemap;

[0013] Step S4: Encapsulate the state information of the key event corresponding to the triggered key event point into an event structure and write it into the perf ring buffer associated with output_map;

[0014] Step S5: The user-space program asynchronously reads the encapsulated events from the perf ring buffer, classifies and aggregates them according to the process and event type, generates delayed terminal logs and histograms, and outputs diagnostic results.

[0015] Among them, the key events corresponding to the multiple key event points that the BPF program attaches to the kernel include the kswapd event, the direct memory reclamation event, the cgroup memory reclamation event, and the memory consolidation event.

[0016] Among them, the features are:

[0017] The key event points corresponding to the kswapd event that the BPF program attaches to the kernel are as follows:

[0018] mm_vmscan_wakeup_kswapd: Triggered when the memory level in a certain area falls below a predetermined value, waking up kswapd, indicating that the system has started asynchronous background memory reclamation; used to determine the start time of memory pressure.

[0019] mm_vmscan_kswapd_wake: Triggered when the kswapd process is actually scheduled to run, indicating that kswapd has really started to perform garbage collection, and is used to measure the kswapd wake-up latency;

[0020] mm_vmscan_kswapd_sleep: Triggered when kswapd enters sleep mode after completing the current garbage collection task, indicating that kswapd has ended the current round of garbage collection. It is used to calculate the single runtime of kswapd and evaluate the garbage collection efficiency.

[0021] The key event points corresponding to direct memory reclamation events that the BPF program attaches to the kernel are as follows:

[0022] mm_vmscan_direct_reclaim_begin: This is triggered when a process fails to allocate memory and triggers synchronous reclamation. It indicates that the application thread is blocked and direct memory reclamation begins. It is used to mark the starting point of latency measurement.

[0023] mm_vmscan_direct_reclaim_end: Triggered when direct memory reclamation is complete, the process continues to execute, indicating that the application thread has resumed running. It is used to mark the end of the delay and calculate the delay of direct memory reclamation.

[0024] The key event points corresponding to cgroup memory reclamation events that the BPF program attaches to the kernel are as follows:

[0025] mm_vmscan_memcg_reclaim_begin: Triggered by cgroup memory pressure, used to distinguish between global reclamation and container-level reclamation, marking the start of container-level reclamation;

[0026] mm_vmscan_memcg_reclaim_end: Triggered when cgroup reclamation ends, indicating that container reclamation has ended, used to calculate reclamation delay;

[0027] The key event points corresponding to the memory consolidation event that the BPF program attaches to the kernel are as follows:

[0028] mm_compaction_begin: This is triggered when the system begins to defragment memory and performs memory compaction. It indicates that the system is attempting to merge free pages to satisfy higher-order allocation and is used to mark the start of the memory compaction delay.

[0029] mm_compaction_end: Triggered when memory compaction ends, indicating that compaction is complete. It is used to calculate compaction time and determine whether it has become a performance bottleneck.

[0030] Specifically, for the kswapd event: when mm_vmscan_wakeup_kswapd is triggered, the obtained status information includes the NUMA node and wake-up timestamp; when mm_vmscan_kswapd_wake is triggered, the obtained status information includes the NUMA node, the actual running timestamp, and the wake-up delay obtained based on the difference between the actual running timestamp and the wake-up timestamp; when mm_vmscan_kswapd_sleep is triggered, the obtained status information includes the NUMA node, sleep timestamp, number of pages requested for reclamation, number of bytes requested for reclamation, number of pages successfully reclamated, number of bytes successfully reclamated, and the single runtime obtained based on the difference between the sleep timestamp and the actual running timestamp.

[0031] For direct memory reclamation events: When mm_vmscan_direct_reclaim_begin is triggered, the obtained status information includes the process name, process PID, and start timestamp; when mm_vmscan_direct_reclaim_end is triggered, the obtained status information includes the process name, process PID, end timestamp, number of pages requested for reclamation, number of bytes requested for reclamation, number of pages successfully reclamated, number of bytes successfully reclamated, and the delay time obtained based on the difference between the end timestamp and the start timestamp.

[0032] For cgroup memory reclamation events: When mm_vmscan_memcg_reclaim_begin is triggered, the obtained status information includes the process name, process PID, and start timestamp; when mm_vmscan_memcg_reclaim_end is triggered, the obtained status information includes the process name, process PID, end timestamp, number of pages requested for reclamation, number of bytes requested for reclamation, number of pages successfully reclamated, number of bytes successfully reclamated, and the delay time obtained based on the difference between the end timestamp and the start timestamp.

[0033] For memory compaction events: When mm_compaction_begin is triggered, the obtained status information includes the process name, process PID, and start timestamp; when mm_compaction_end is triggered, the obtained status information includes the process name, process PID, end timestamp, number of pages requested for compaction, number of bytes requested for compaction, number of pages successfully compacted, number of bytes successfully compacted, compaction status, and delay time based on the difference between the end timestamp and the start timestamp.

[0034] Specifically, the state map maintains a memlatency_kswapd map, a memlatency_direct_reclaim map, a memlatency_memcg_reclaim map, and a memlatency_compaction map for kswapd events, direct memory reclamation events, cgroup memory reclamation events, and memory compaction events, respectively, and records the state information of kswapd events, direct memory reclamation events, cgroup memory reclamation events, and memory compaction events in the form of key-value pairs.

[0035] Specifically, for kswapd events, the memlatency_kswapd map uses NUMA nodes as keys;

[0036] For direct memory reclamation events, cgroup memory reclamation events, and memory compaction events, the process PID is used as the key in the memlatency_direct_reclaim map, memlatency_memcg_reclaim map, and memlatency_compaction map.

[0037] In this process, while loading the BPF program into the kernel subsystem and attaching the BPF program to multiple key event points on the kernel, the system also receives user-defined monitoring parameters to control the monitoring behavior and data acquisition cycle of the BPF program.

[0038] In step S5, the user-space program calls dump_ after the monitoring period ends. The _trace() function iterates through all state maps in the perf ring buffer, asynchronously reading the encapsulated events.

[0039] In step S5, the user-mode program outputs the diagnostic results in the form of a log file.

[0040] Specifically, for kswapd events, direct memory reclamation events, cgroup memory reclamation events, and compaction events, user-space programs output diagnostic results in the form of kswapd.log, direct_reclaim.log, cgroup_reclaim.log, and compaction.log, respectively.

[0041] This invention, through systematic architecture design, combines the state preservation capability and kernel context access capability of eBPF with the event model of the virtual memory subsystem to construct a closed-loop monitoring system oriented towards diagnostic objectives, achieving high-precision and high-reliability memory operation latency measurement. Attached Figure Description

[0042] Figure 1 The system architecture diagram of the progressive memory latency monitoring and diagnosis method for virtual memory subsystems of the present invention is shown.

[0043] Figure 2 The flowchart of the progressive memory latency monitoring and diagnosis method for virtual memory subsystems of the present invention is shown below.

[0044] Figure 3 Another system architecture diagram of the progressive memory latency monitoring and diagnosis method for virtual memory subsystem of the present invention.

[0045] Figure 4 The critical path signaling diagram of an embodiment of the progressive memory latency monitoring and diagnosis method for virtual memory subsystems of the present invention. Detailed Implementation

[0046] To gain a better understanding of the technical solution and beneficial effects of the present invention, the technical solution of the present invention and its beneficial effects are described in detail below with reference to the accompanying drawings.

[0047] To address the shortcomings of existing technologies where memory pressure leads to slow kswapd response, increased memory pressure causing frequent direct reclaims, and compaction failures resulting in application lag, this progressive memory pressure propagation chain lacks systematic modeling, hindering link-level performance attribution. This invention provides a progressive memory latency monitoring and diagnosis method for the virtual memory subsystem. Its core objectives are: to accurately capture and calculate the begin / end events of key memory operations such as kswapd, direct reclaim / memcg reclaim, and compaction in kernel mode; to maintain context state through BPF Maps to ensure accurate event matching; to achieve multi-dimensional data aggregation and diagnostic report generation in user mode, supporting root cause analysis; and to construct a progressive path model from background reclamation to direct reclamation to memory compaction, enabling link-based diagnosis of memory latency.

[0048] Please combine Figures 1 to 3 As shown, the progressive memory latency monitoring and diagnosis method for virtual memory subsystems provided by the present invention includes the following steps S1-S5.

[0049] Step S1: The user-space diagnostic tool starts, loads its compiled BPF program into the kernel subsystem, and mounts the BPF program to multiple tracepoints on the kernel. It receives monitoring parameters of the BPF program specified by the user through the command line or configuration file to control the monitoring behavior and data acquisition cycle of the BPF program. If no monitoring parameters are passed, the system's built-in monitoring parameter configuration is used by default.

[0050] Specifically, user-space tools encapsulate BPF system calls through the libbpf library (corresponding to...). Figure 3 The `libbpf+BPFsyscall` library loads and manages BPF programs, controlling their monitoring behavior and data acquisition cycles to monitor and interact with memory latency events. BPF system calls serve as a crucial bridge for communication between user space and kernel space. The tracepoints attached to BPF programs involve kswapd events, direct memory reclamation events, cgroup memory reclamation events, and memory consolidation events within the kernel subsystem. When a kernel event is triggered, the BPF program retrieves the status information of these key events from the tracepoint.

[0051] Please combine Figure 2 and Figure 3 As shown, the tracepoints involved in each key event are as follows:

[0052] Kswapd incident:

[0053] mm_vmscan_wakeup_kswapd: Triggered when the memory level of a zone falls below min, waking up kswapd, indicating that the system needs to start asynchronous background memory reclamation; can be used to determine when memory pressure begins;

[0054] mm_vmscan_kswapd_wake: Triggered when the kswapd process is actually scheduled to run, kswapd actually begins to perform garbage collection; the kswapd wake-up latency (from wakeup to wake) can be measured. This latency is caused by scheduling latency, CPU blocking, etc. If the latency is large, it indicates that the system load is high and kswapd cannot respond in time.

[0055] mm_vmscan_kswapd_sleep: Triggered when kswapd enters sleep mode after completing the current garbage collection task, indicating that kswapd has ended the current round of garbage collection. It can be used to calculate the runtime of a single kswapd session and evaluate the garbage collection efficiency.

[0056] Direct memory reclamation:

[0057] mm_vmscan_direct_reclaim_begin: This is triggered when a process fails to allocate memory and triggers synchronous reclamation, indicating that the application thread is blocked and direct memory reclamation begins; it can be used to mark the starting point of latency measurement.

[0058] mm_vmscan_direct_reclaim_end: Triggered when direct memory reclamation is complete, the process continues to execute, indicating that the application thread has resumed running, can mark the end of the delay, and calculate the delay of direct memory reclamation;

[0059] cgroup memory reclamation:

[0060] mm_vmscan_memcg_reclaim_begin: Triggered by cgroup memory pressure, it can be used to distinguish between global reclamation and container-level reclamation, and mark the starting point of container-level reclamation;

[0061] mm_vmscan_memcg_reclaim_end: Triggered when cgroup reclamation ends, indicating that container reclamation is complete, and reclamation delay can be calculated;

[0062] Memory consolidation events:

[0063] mm_compaction_begin: This is triggered when the system begins to defragment memory and performs memory compaction. It indicates that the system is attempting to merge free pages to satisfy higher-order allocations; it can also mark the start point of memory compaction delay.

[0064] mm_compaction_end: Triggered when memory compaction ends, indicating that compaction is complete (whether successful or not). The compaction time can be calculated to determine whether it has become a performance bottleneck.

[0065] Therefore, the monitoring points constructed in this invention involve background reclamation latency (kswapd wake-up latency, runtime) - direct memory reclamation latency - cgroup-level memory reclamation latency (memcg reclaim) - memory compaction latency. By establishing a progressive diagnostic model from background asynchronous reclamation to foreground blocking reclamation, it supports accurate attribution of memory performance bottlenecks. Monitoring points are deployed separately to achieve full-link latency tracing and coverage of the entire lifecycle of memory operations. It supports unified monitoring and correlation analysis of multiple events, realizing the link-based root cause localization from "background reclamation failure" to "frequent direct reclamation" to "application lag". It completes the link-based diagnosis from "phenomenon" to "root cause", improves diagnostic accuracy, forms a closed-loop event flow, and breaks through the limitations of traditional tools that analyze single events in isolation. Moreover, it does not require modification of the kernel or application code, achieving non-intrusive, low-overhead monitoring with strong compatibility and convenient deployment.

[0066] Step S2: Initialize the BPF Map hash table - state map - for storing information such as timestamps and process states of key events; initialize the configuration table config_map for receiving filtered events set by the user, such as monitoring only processes with specific PIDs or specific cgroups; initialize the perf ringbuffer associated with output_map for batch exporting delayed events to user space (detailed below).

[0067] `state_map` is a specific instance of a BPF Map. A BPF Map is a general key-value storage mechanism provided by eBPF for sharing data between the kernel and user space; while `state_map` is the variable name of a specific BPF Map.

[0068] Step S3: When the critical event point is triggered (corresponding to...) Figure 1 "Kernel event triggering" Figure 2 "Looping wait for event triggering" and Figure 3The process involves retrieving the state information of key events triggered by virtual memory events and writing them into a statemap. For kswapd events, direct reclaim events, cgroup reclaim events, and memory compaction events, the statemap maintains separate memlatency_kswapd, memlatency_direct_reclaim, memlatency_memcg_reclaim, and memlatency_compaction maps, respectively, recording the state information of each event in key-value pairs. All data is calculated in kernel space, avoiding frequent data interaction between kernel and user space.

[0069] Regarding the kswapd event: When mm_vmscan_wakeup_kswapd is triggered, the acquired state information includes the NUMA node and the wakeup timestamp wakeup_ts. Specifically, the NUMA node is written to the memlatency_kswapd map of the state_map as the key tgid in the key-value pair, marking the start of this kswapd event; When mm_vmscan_kswapd_wake is triggered, the acquired state information includes the NUMA node and the actual runtime timestamp wake_ts, as well as the wakeup delay obtained based on the difference between the actual runtime timestamp wake_ts and the wakeup timestamp wakeup_ts. Specifically, based on the NUMA node, the wakeup timestamp wakeup_ts corresponding to the key tgid of the NUMA node is found in the state_map, and the wakeup delay (wake_ts-wakeup_ts) is calculated; mm_vmsc When an_kswapd_sleep is triggered, the obtained state information includes the NUMA node, sleep timestamp sleep_ts, number of pages requested for reclamation, number of bytes requested for reclamation, number of pages successfully reclamped, number of bytes successfully reclamped, and the single runtime obtained based on the difference between the sleep timestamp sleep_ts and the actual runtime timestamp wake_ts. Specifically, based on the NUMA node, the actual runtime timestamp wake_ts corresponding to the key tgid of the NUMA node is found in the state_map, and the single runtime (wake_ts-sleep_ts) is calculated.

[0070] Similarly, for direct memory reclamation events: when `mm_vmscan_direct_reclaim_begin` is triggered, the obtained status information includes the process name, process PID, and start timestamp `begin_ts`. Specifically, the process PID is used as the key `tgid` in the key-value pair and written into the `memlatency_direct_reclaim map` of the `state_map`. When `mm_vmscan_direct_reclaim_end` is triggered, the obtained status information includes the process name, process PID, end timestamp `end_ts`, number of pages requested for reclamation, number of bytes requested for reclamation, number of pages successfully reclamped, number of bytes successfully reclamped, and the delay time obtained based on the difference between the end timestamp and the start timestamp. Specifically, based on the process PID, the start timestamp `begin_ts` of the key `tgid` corresponding to the process PID is found in the `state_map`, and the delay time `(end_ts - begin_ts)` is calculated.

[0071] Similarly, for cgroup memory reclamation events: when `mm_vmscan_memcg_reclaim_begin` is triggered, the obtained status information includes the process name, process PID, and start timestamp `begin_ts`. Specifically, the process PID is used as the key `tgid` in the key-value pair and written to the `memlatency_memcg_reclaim` map of the `state_map`. When `mm_vmscan_memcg_reclaim_end` is triggered, the obtained status information includes the process name, process PID, end timestamp `end_ts`, number of pages requested for reclamation, number of bytes requested for reclamation, number of pages successfully reclamped, number of bytes successfully reclamped, and the delay time obtained based on the difference between the end timestamp and the start timestamp. Specifically, based on the process PID, the start timestamp `begin_ts` of the key `tgid` corresponding to the process PID is found in the `state_map`, and the delay time `(end_ts - begin_ts)` is calculated.

[0072] Similarly, for memory compaction events: when mm_compaction_begin is triggered, the obtained status information includes the process name, process PID, and start timestamp begin_ts. Specifically, the process PID is used as the key tgid in the key-value pair and written into the memlatency_compaction map of the state_map. When mm_compaction_end is triggered, the obtained status information includes the process name, process PID, end timestamp end_ts, number of pages requested for compaction, number of bytes requested for compaction, number of pages successfully compacted, number of bytes successfully compacted, compaction status, and delay time obtained based on the difference between the end timestamp and the start timestamp. Specifically, based on the process PID, the start timestamp begin_ts of the key tgid corresponding to the process PID is found in the state_map, and the delay time (end_ts - begin_ts) is calculated.

[0073] Therefore, this invention maintains the timestamps of begin or wake events and other process state information in the kernel to ensure that when end or sleep events are triggered, the latency can be accurately calculated directly in the kernel and performance metrics can be updated (i.e., the number of pages requested for reclamation / regularization, the number of bytes requested for reclamation / regularization, the number of pages successfully reclamated / regularized, the number of bytes successfully reclamated / regularized, and the timestamps when each key event point is triggered). All data is exported in batches by user space only after the monitoring period ends, avoiding the failure of reconstruction caused by the state being returned to user space. By maintaining cross-event states through BPF map, event loss and time drift caused by high concurrency or user space log parsing misalignment are eliminated. This avoids time errors introduced by user space tools due to scheduling delays, log sampling intervals, and other factors, ensuring that the latency calculation accuracy reaches the nanosecond level. The hash structure of BPF map also ensures lookup efficiency and maintains low overhead even in high concurrency scenarios, solving the problem of "event mismatch leading to artificially high latency or loss" in traditional methods. Experimental data shows that under the pressure of tens of thousands of direct reclaims per second, the present invention can still maintain an event matching success rate of over 99.9%, while the matching rate of the ftrace-based method is less than 85%.

[0074] Meanwhile, this invention not only collects latency time, but also synchronously records performance indicators such as the number of pages requested for reclamation, the number of bytes requested for reclamation, the number of pages successfully reclamated, the number of bytes successfully reclamated, and the normalization results, supporting comprehensive diagnosis.

[0075] Meanwhile, this invention employs a differentiated key design strategy based on the semantic characteristics of different memory events: for memory operations triggered by user processes (such as direct reclaim and compaction), the process PID is used as the key in the BPF Map to achieve process-level latency attribution; while for system-level background threads (such as kswapd), the NUMA node ID is used as the key to accurately reflect the pressure status of each memory node, precisely identify uneven memory load among NUMA nodes, and provide data support for resource scheduling optimization. This key selection mechanism balances monitoring accuracy and system overhead, supporting progressive diagnosis of memory latency.

[0076] Finally, this invention treats tracepoints such as mm_vmscan_kswapd_wake / sleep, mm_vmscan_direct_reclaim_begin / end, m_vmscan_memcg_reclaim_begin / end, and mm_compaction_begin / end as a logically related performance chain. It designs independent BPFMaps for different memory operation paths, meaning that the Map can optimize its data structure for its monitored objects (e.g., the memlatency_kswapd map contains a delay_wakeup_to_wake field, while the memlatency_direct_reclaim map and memlatency_memcg_reclaim map contain pages_requested). A multi-Map structure is used to store the states of different paths, and the keys of each Map involve semantically meaningful dimensions (e.g., process PID, NUMA node ID), making data aggregation more efficient. It supports data isolation and aggregation by process, node, cgroup, etc. (e.g., it can quickly calculate the direct reclaim latency of a certain PID, or the kswapd latency of a certain NUMA node). (Average runtime), and the key-value design of Maps inherently supports multi-dimensional slicing analysis, enabling diagnostic output "by process," "by node," and "by cgroup" without additional parsing. It supports fine-grained performance analysis (such as identifying bottlenecks in specific processes or NUMA nodes), meeting the diagnostic needs of complex scenarios such as databases and containerization. It provides data support for multi-log joint analysis of user-space tools, allowing user-space programs to traverse Maps of different structures as needed, generating structured text logs and forming an observability loop. In Kubernetes scenarios, operations personnel can directly locate specific Pods based on the PID field in cgroup_reclaim.log and judge their memory reclamation efficiency by combining the pages_reclaimed / pages_requested ratio, without relying on additional metrics exposed during container runtime.

[0077] Step S4: Encapsulate the state information of the key event corresponding to the triggered key event point into an event structure, and call bpf_perf_event_output() to write the event structure into the perf ring buffer associated with output_map (corresponding to...). Figure 1 "Send data" and Figure 2 After writing the event (delayed event write), the record corresponding to the unique key tgid of the event structure is deleted from the state_map, and resources are released.

[0078] For the kswapd event, its status information includes: NUMA node, wakeup timestamp wakeup_ts when mm_vmscan_wakeup_kswapd is triggered, actual runtime timestamp wake_ts when mm_vmscan_kswapd_wake is triggered, sleep timestamp sleep_ts when mm_vmscan_kswapd_sleep is triggered, number of pages requested to be reclaimed, number of bytes requested to be reclaimed, number of pages successfully reclaimed, number of bytes successfully reclaimed, and single runtime duration;

[0079] For direct memory reclamation events, the status information includes: process name, process PID, start timestamp begin_ts when mm_vmscan_direct_reclaim_begin is triggered, end timestamp end_ts when mm_vmscan_direct_reclaim_end is triggered, number of pages requested to be reclaimed, number of bytes requested to be reclaimed, number of pages successfully reclaimed, number of bytes successfully reclaimed, and delay time.

[0080] For cgroup memory reclamation events, the status information includes: process name, process PID, start timestamp begin_ts when mm_vmscan_memcg_reclaim_begin is triggered, end timestamp end_ts when mm_vmscan_memcg_reclaim_end is triggered, number of pages requested to be reclaimed, number of bytes requested to be reclaimed, number of pages successfully reclaimed, number of bytes successfully reclaimed, and delay time.

[0081] For memory compaction events, the status information includes: process name, process PID, start timestamp begin_ts when mm_compaction_begin is triggered, end timestamp end_ts when mm_compaction_end is triggered, number of pages requested to be reclaimed, number of bytes requested to be reclaimed, number of pages successfully compacted, number of bytes successfully compacted, compaction status, and delay time.

[0082] Step S5: After the monitoring period ends, the user-space program calls dump_ The _trace() function continuously iterates through all state maps in the perf ring buffer using poll(), asynchronously reading the encapsulated events to avoid high-frequency interrupt overhead. Upon receiving an event, it categorizes and aggregates it according to the process and event type, generates delayed terminal logs and histograms, outputs diagnostic results / diagnostic reports, and performs automated analysis and optimization based on the diagnostic results and monitoring data in the event structure.

[0083] Here, output_map is a Map handle, while perf ring buffer is the underlying transmission channel. perf ring buffer itself is a ring buffer provided by the kernel perf subsystem, one for each CPU, managed by output_map, and the bpf_perf_event_output() function is a BPF helper function used to write data to the current CPU's ring buffer.

[0084] In this invention, the user-space program outputs diagnostic results in the form of log files. For kswapd events, direct memory reclamation events, cgroup memory reclamation events, and memory compaction events, the user-space program outputs diagnostic results in the forms of kswapd.log, direct_reclaim.log, cgroup_reclaim.log, and compaction.log, respectively.

[0085] A specific application scenario is as follows: A database instance experiences latency spikes of around 100ms during peak periods. By using this invention for monitoring and analysis, and enabling compaction latency monitoring, it was found that the `status=COMPACT_CONTINUE` value frequently appeared in the `compaction_end` event, with each instance taking >50ms. Combined with high direct reclaim latency, this was determined to be due to severe memory fragmentation. Diagnosis conclusion: Memory fragmentation causes compaction to be executed multiple times, resulting in latency spikes. Optimization suggestion: Enable transparent hugepage or adjust `zone_reclaim_mode`.

[0086] Another specific application scenario is as follows: When system memory pressure increases, kswapd should wake up and reclaim pages in a timely manner to alleviate the pressure. If memlatency_kswapd shows that the kswapd wakeup delay (wakeup → wake) is too long or the runtime is insufficient, it indicates that the background reclamation capability has decreased. At this time, if the number of direct_reclaims in memlatency_reclaim increases significantly and the delay increases, a causal chain of "kswapd failure → frequent direct reclaims → application delay" can be established. If a high compaction failure rate is further found in memlatency_compaction, it can be inferred that memory fragmentation is the root cause.

[0087] Therefore, the analysis mechanism based on path collaborative modeling in this invention forms a logical closed loop from previously isolated performance metrics, greatly improving the ability to locate root causes. In the case of database latency spikes, this invention successfully identified the complete chain of "slow response of Node 1kswapd causing the local process to fallback to direct reclaim, which further triggers compaction failure and causes query lag," while traditional tools can only report "long reclaim time" and cannot trace the upstream cause.

[0088] Furthermore, in this invention, after the kernel mode completes the matching and delay calculation of all events, the user mode only reads the BPF Map once at the end of the monitoring period; it uses an efficient hash table structure and a zero-copy mechanism. Compared to traditional methods that require frequent transmission of raw event logs from the kernel to the user mode, resulting in a large number of system calls and context switches, this invention only performs a Map update operation when the end event or sleep is triggered, and does not involve cross-privilege level data transmission. During monitoring, CPU overhead mainly comes from tracepoint callbacks and Map writes, with an average load of <3% in actual tests; the data export stage uses batch reading, avoiding the continuous resource occupation caused by real-time streaming processing; the overall implementation is low-intrusive and low-overhead, suitable for long-term deployment in production environments. Running monitoring continuously for 1 hour on a 64-core server, the average system load increased by less than 0.1, while under the same conditions, using perf record -e caused the load to increase by more than 1.5.

[0089] Therefore, this invention realizes progressive path tracing, stateful delay aggregation, and enhanced diagnostic functions. Compared with existing eBPF tools, it has the following significant differences: (1) Progressive path tracing: It not only records individual events, but also establishes a full-link delay tracing model from kswapd to direct reclaim to compaction; (2) Stateful aggregation analysis: It maintains the event context state (such as begin / end timestamps) through BPF Map to achieve accurate matching and delay calculation; (3) Low-overhead batch export: It adopts a batch export mechanism to avoid high-frequency event reporting and reduce system overhead; (4) Enhanced diagnostic functions: It provides advanced diagnostic functions such as delay analysis logs, wake-up delay analysis, and compaction success rate statistics to improve operability, is suitable for cloud-native environments, and supports performance isolation and responsibility definition in multi-tenant scenarios.

[0090] Figure 4This is a critical path signaling diagram involved in a specific embodiment of the present invention. In this embodiment, the root cause of memory latency in a high-concurrency database service on a NUMA architecture server is located: an OLTP database service deployed on a NUMA architecture server frequently experiences query latency spikes (>100ms) during peak business periods. Initial investigation ruled out network and disk I / O bottlenecks, and the synchronous reclamation behavior of the virtual memory subsystem was suspected to be related. The judgment logic for insufficient reclamation is that the memory pressure gradually increases due to insufficient kswapd reclamation, frequent triggering of direct reclaim, triggering of cgroup reclaim, and triggering of compaction, indicating insufficient reclamation. Figure 4 In this context, the runtime of dire_reclaim and the runtime of cgroupreclaim refer to the delay time of direct memory reclamation events and cgroup memory reclamation events, respectively.

[0091] based on Figure 4 The path signaling graph is used to perform a linkage analysis of direct reclamation and regularization events. If there are cases of background memory reclamation failure and severe memory fragmentation, it indicates that the memory reclamation failure is caused by severe fragmentation.

[0092] The logic for determining if background memory reclamation has failed involves the following three scenarios:

[0093] Scenario 1: If direct_reclaim_begin occurs outside the kswapd running window, it indicates that kswapd is not running or is asleep. The significance of this diagnosis is that background garbage collection failed to respond to memory pressure in a timely manner, forcing user processes to trigger direct memory reclamation.

[0094] Scenario 2: When direct_reclaim_begin occurs, kswapd is running, but direct_reclaim_duration is very long, indicating that kswapd is not reclaiming enough and memory is still tight. The diagnostic significance is that the background reclamation efficiency is low.

[0095] Scenario 3: When direct_reclaim_begin occurs, kswapd is running, but kswapd_duration is very short, indicating that kswapd was woken up but quickly went to sleep. The short kswapd_duration means that kswapd was woken up but quickly went to sleep, and the reclamation goal was not achieved.

[0096] If compaction_end returns COMPACT_CONTINUE or COMPACT_NO_SUITABLE_PAGE at this point, it indicates severe memory fragmentation.

[0097] After the diagnosis is completed, the user-space diagnostic tool aggregates direct reclaim latency, compaction success rate, and status information according to the process PID and outputs it to the log file.

[0098] Although the present invention has been described using the above preferred embodiments, it is not intended to limit the scope of protection of the present invention. Any changes and modifications made by those skilled in the art to the above embodiments without departing from the spirit and scope of the present invention shall still fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be defined by the claims.

Claims

1. A progressive memory latency monitoring and diagnosis method for virtual memory subsystems, characterized in that, include: Step S1: Load the BPF program into the kernel subsystem and attach the BPF program to multiple critical event points on the kernel; Step S2: Initialize the hash table - state map - for recording key event states of users; Step S3: When a critical event is triggered, obtain the state information of the corresponding critical event and write it into the state map; Step S4: Encapsulate the state information of the key event corresponding to the triggered key event point into an event structure and write it into the perf ring buffer associated with output_map; Step S5: The user-space program asynchronously reads the encapsulated events from the perf ring buffer, classifies and aggregates them according to process and event type, generates delayed terminal logs and histograms, and outputs diagnostic results; The key events corresponding to the multiple critical event points that the BPF program attaches to the kernel include the kswapd event, the direct memory reclamation event, the cgroup memory reclamation event, and the memory consolidation event. While loading the BPF program into the kernel subsystem and attaching the BPF program to multiple critical event points on the kernel, it receives user-defined monitoring parameters to control the monitoring behavior and data acquisition cycle of the BPF program.

2. The progressive memory latency monitoring and diagnosis method for virtual memory subsystems as described in claim 1, characterized in that: The key event points corresponding to the kswapd event that the BPF program attaches to the kernel are as follows: mm_vmscan_wakeup_kswapd: Triggered when the memory level in a certain area falls below a predetermined value, waking up kswapd, indicating that the system has started asynchronous background memory reclamation; used to determine the start time of memory pressure. mm_vmscan_kswapd_wake: Triggered when the kswapd process is actually scheduled to run, indicating that kswapd has really started to perform garbage collection, and is used to measure the kswapd wake-up latency; mm_vmscan_kswapd_sleep: Triggered when kswapd enters sleep mode after completing the current garbage collection task, indicating that kswapd has ended the current round of garbage collection. It is used to calculate the single runtime of kswapd and evaluate the garbage collection efficiency. The key event points corresponding to direct memory reclamation events that the BPF program attaches to the kernel are as follows: mm_vmscan_direct_reclaim_begin: This is triggered when a process fails to allocate memory and triggers synchronous reclamation. It indicates that the application thread is blocked and direct memory reclamation begins. It is used to mark the starting point of latency measurement. mm_vmscan_direct_reclaim_end: Triggered when direct memory reclamation is complete, the process continues to execute, indicating that the application thread has resumed running. It is used to mark the end of the delay and calculate the delay of direct memory reclamation. The key event points corresponding to cgroup memory reclamation events that the BPF program attaches to the kernel are as follows: mm_vmscan_memcg_reclaim_begin: Triggered by cgroup memory pressure, used to distinguish between global reclamation and container-level reclamation, marking the start of container-level reclamation; mm_vmscan_memcg_reclaim_end: Triggered when cgroup reclamation ends, indicating that container reclamation has ended, used to calculate reclamation delay; The key event points corresponding to the memory consolidation event that the BPF program attaches to the kernel are as follows: mm_compaction_begin: This is triggered when the system begins to defragment memory and performs memory compaction. It indicates that the system is attempting to merge free pages to satisfy higher-order allocation and is used to mark the start of the memory compaction delay. mm_compaction_end: Triggered when memory compaction ends, indicating that compaction is complete. It is used to calculate compaction time and determine whether it has become a performance bottleneck.

3. The progressive memory latency monitoring and diagnosis method for virtual memory subsystems as described in claim 2, characterized in that: For the kswapd event: When mm_vmscan_wakeup_kswapd is triggered, the obtained status information includes the NUMA node and wake-up timestamp; when mm_vmscan_kswapd_wake is triggered, the obtained status information includes the NUMA node, the actual running timestamp, and the wake-up delay obtained based on the difference between the actual running timestamp and the wake-up timestamp; when mm_vmscan_kswapd_sleep is triggered, the obtained status information includes the NUMA node, sleep timestamp, number of pages requested for reclamation, number of bytes requested for reclamation, number of pages successfully reclamated, number of bytes successfully reclamated, and the single runtime obtained based on the difference between the sleep timestamp and the actual running timestamp. For direct memory reclamation events: When mm_vmscan_direct_reclaim_begin is triggered, the obtained status information includes the process name, process PID, and start timestamp; when mm_vmscan_direct_reclaim_end is triggered, the obtained status information includes the process name, process PID, end timestamp, number of pages requested for reclamation, number of bytes requested for reclamation, number of pages successfully reclamated, number of bytes successfully reclamated, and the delay time obtained based on the difference between the end timestamp and the start timestamp. For cgroup memory reclamation events: When mm_vmscan_memcg_reclaim_begin is triggered, the obtained status information includes the process name, process PID, and start timestamp; when mm_vmscan_memcg_reclaim_end is triggered, the obtained status information includes the process name, process PID, end timestamp, number of pages requested for reclamation, number of bytes requested for reclamation, number of pages successfully reclamated, number of bytes successfully reclamated, and the delay time obtained based on the difference between the end timestamp and the start timestamp. For memory compaction events: When mm_compaction_begin is triggered, the obtained status information includes the process name, process PID, and start timestamp; when mm_compaction_end is triggered, the obtained status information includes the process name, process PID, end timestamp, number of pages requested for compaction, number of bytes requested for compaction, number of pages successfully compacted, number of bytes successfully compacted, compaction status, and delay time based on the difference between the end timestamp and the start timestamp.

4. The progressive memory latency monitoring and diagnosis method for virtual memory subsystems as described in claim 3, characterized in that: The state maps maintain memlatency_kswapd map, memlatency_direct_reclaim map, memlatency_memcg_reclaim map, and memlatency_compaction map for kswapd events, direct memory reclamation events, cgroup memory reclamation events, and memory compaction events, respectively, and record the state information of kswapd events, direct memory reclamation events, cgroup memory reclamation events, and memory compaction events in the form of key-value pairs.

5. The progressive memory latency monitoring and diagnosis method for virtual memory subsystems as described in claim 4, characterized in that: For kswapd events, the memlatency_kswapd map uses NUMA nodes as keys; For direct memory reclamation events, cgroup memory reclamation events, and memory compaction events, the process PID is used as the key in the memlatency_direct_reclaim map, memlatency_memcg_reclaim map, and memlatency_compaction map.

6. The progressive memory latency monitoring and diagnosis method for virtual memory subsystems as described in claim 1, characterized in that: In step S5, after the monitoring period ends, the user-mode program calls dump_ The _trace() function iterates through all state maps in the perf ring buffer, asynchronously reading the encapsulated events.

7. The progressive memory latency monitoring and diagnosis method for virtual memory subsystems as described in claim 1, characterized in that: In step S5, the user-mode program outputs the diagnostic results in the form of a log file.

8. The progressive memory latency monitoring and diagnosis method for virtual memory subsystems as described in claim 7, characterized in that: For kswapd events, direct_reclaim events, cgroup memory reclamation events, and compaction events, user-space programs output diagnostic results in the form of kswapd.log, direct_reclaim.log, cgroup_reclaim.log, and compaction.log, respectively.

Images