A method and system for Linux kernel OOM system collection, snapshot and persistence

Abstract

This invention discloses a method and system for collecting, snapshotting, and persisting Linux kernel OOM (Out of Memory) data. Belonging to the field of computer operating system kernel technology, its operation steps are as follows: When the Linux kernel runs out of memory and triggers the OOM Killer, its default process termination process is first intercepted; then, detailed information of the selected process and a snapshot of the system's global memory state are collected; next, the collected structured data is forcibly written to a pre-reserved persistent storage area with high priority and low memory allocation; finally, the original process termination operation is restored and executed. This invention can reliably save complete fault scene information at the moment OOM occurs, forming a black box record, overcoming the volatility of traditional logs, providing a solid data foundation for accurate root cause analysis and system optimization afterward, and significantly improving the maintainability and reliability of the system.

Description

Technical Field

[0001] This invention belongs to the field of computer operating system kernel technology, and relates to a method and system for collecting, snapshotting and persisting Linux kernel OOM system data; specifically, it relates to a method, system and apparatus for collecting system and process information, taking system state snapshots and forcibly persisting them in the Linux kernel Out-Of-Memory (OOM) handling mechanism. Background Technology

[0002] As the core of servers, cloud computing, and embedded devices, the stability of the Linux operating system is crucial. When the system's available memory is severely insufficient, the kernel's memory management subsystem triggers the memory exhaustion killer mechanism, also known as OOMKiller. The core task of this mechanism is to select one or more processes to terminate, thereby freeing up memory and allowing the system to return to normal.

[0003] However, existing OOM Killer mechanisms have a significant drawback: they lack a highly reliable process to save the "failure scene" information that led to the OOM before executing the process termination operation. This poses a huge challenge to system administrators for post-incident root cause analysis, specifically manifested in the following ways: 1. Loss of critical information: Detailed memory usage information of the terminated process (such as anonymous memory-resident set anon-rss, file cache memory-resident set file-rss, etc.), its resource constraint context in the control group (Cgroup), and the overall system memory state (such as MemInfo, Slab cache) are immediately lost after the process is killed, making it difficult to reconstruct the full picture of the incident; 2. Reliance on volatile logs, resulting in poor reliability: Existing technologies mainly rely on system logs (such as dmesg). However, these logs are usually stored in memory buffers or ordinary file systems. Under extreme memory pressure, the logging system itself may fail to log due to the inability to allocate memory; if OOM causes the system to crash or restart, these logs stored on volatile media will be permanently lost, making analysis impossible; 3. Low analysis efficiency and lack of basis for optimization: Due to the lack of accurate on-site data, analysts can only rely on experience to speculate on the root cause of OOM, making it difficult to accurately adjust application memory parameters, Cgroup configurations or kernel parameters (such as vm.overcommit_memory), which may cause the problem to recur. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a method for collecting, snapshotting, and persisting Linux kernel OOM system data before the execution of the OOM Killer, ensuring the generation of an unalterable fault scene record, and greatly improving the system's problem localization and maintenance capabilities.

[0005] Another objective of this invention is to provide a system for implementing a method for system information acquisition, snapshotting, and persistence in Linux kernel OOM processing.

[0006] The technical solution of this invention is: the method for collecting, snapshotting, and persisting Linux kernel OOM systems according to this invention includes the following steps: Step (1) Triggering and intercepting; Step (2) Information collection; Step (3) Force persistence; Step (4) continues.

[0007] Furthermore, the triggering and interception described in step (1) are as follows: when the Linux kernel's memory management subsystem determines that the system memory is exhausted and decides to call the OOM Killer, it first intercepts its default process selection and termination process.

[0008] Furthermore, the detailed information of the intercepted process includes: process identifier (PID), process name, OOM weight score, virtual memory and physical memory usage details, page table size, and the control group (Cgroup) information to which the process belongs.

[0009] Furthermore, the information collection mentioned in step (2) specifically involves: in the interception state, performing multi-dimensional information collection, namely, collecting detailed information of the process selected by the OOM Killer, and synchronously collecting a snapshot of the system's global memory state; Specifically: 2.1) Collection of detailed information of the selected process: Collect detailed information of the process selected by the OOM Killer, including process identifier (PID), process name, OOM weight score (oom_score and oom_score_adj), total virtual memory (total-vm), physical memory resident set details (including anon-rss, file-rss, shmem-rss), page table size (pgtables), and the control group (Cgroup) to which the process belongs and resource limit information; 2.2) System Global Memory Status Snapshot Acquisition: Synchronously collect system-wide memory status snapshots. These snapshots include detailed information (Mem-Info) for each memory node, Slab cache usage statistics, swap space usage status, and I / O pressure. Furthermore, the snapshot of the global memory state includes: node memory information Mem-Info, Slab cache usage, and swap space status.

[0010] Furthermore, the forced persistence mentioned in step (3) specifically means: directly writing the process information and system state snapshot (structured data) collected in step (2) into the pre-configured persistent storage area through a high-priority, low-memory allocation method; The persistent storage area includes non-volatile memory (NVM), protected disk partitions, or specially marked files.

[0011] Furthermore, the process described in step (4) is as follows: after confirming that the information has been successfully written to the persistent storage area, the interception of the OOM Killer is lifted, allowing it to continue executing the original process termination process and release memory.

[0012] Furthermore, a system for implementing the method includes: Interception module: Integrated into the Linux kernel memory management subsystem, it is used to listen for OOM events and intercept the default process when OOMKiller is triggered; Information acquisition module: connected to the interception module, used to perform detailed information acquisition of the selected process and system global memory state snapshot acquisition after receiving the interception signal; Persistence module: Connected to the information acquisition module, configured with high priority, used to force the acquired information to be written into the persistent storage area.

[0013] Furthermore, an apparatus includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method.

[0014] Furthermore, a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described thereon.

[0015] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: 1. Traceable Fault Scene: It retains a complete and structured "black box" record for each OOM event, enabling accurate reconstruction of the system state at the moment of the fault, greatly improving system maintainability; 2. Highly Reliable Information Storage: Through high-priority design, a minimized memory allocation strategy, and writing to protected persistent media, it ensures that fault information can be completely preserved even under the most extreme memory pressure or after system crash, overcoming the susceptibility of traditional logs; 3. Precise Operation and Maintenance Optimization: System administrators can accurately locate the source of memory leaks and configure unreasonable Cgroup or kernel parameters based on detailed on-site data, thereby taking effective measures to prevent OOM recurrence and improving the long-term stability and reliability of the system. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the module architecture of the present invention. Detailed Implementation

[0017] The specific technical solution of the present invention will be further described in detail below with reference to specific examples.

[0018] Example 1: Implementation using kernel modules The operation steps of a loadable Linux kernel module (LKM) described in this embodiment are as follows: Step 1, Initialization: In the module initialization function, a notification callback function oom_notifier_call is registered with the kernel's OOM notification chain through the register_oom_notifier() function; at the same time, a file is created in the file system (such as / sys / kernel / oom_blackbox) or a memory-mapped NVM area is reserved as a persistent storage area in the early stage of system startup or when the module is loaded. Step 2, Triggering and Interception: When an OOM occurs, the kernel calls all registered notifier functions oom_notifier_call; in this function, it does not immediately return NOTIFY_OK, but begins to execute subsequent steps, which in effect "intercepts" the immediate execution of the OOM Killer; Step 3, Information Collection: Step 301, Process Information: The callback function parameter provides a pointer to the task_struct structure of the selected process; by traversing this structure, you can obtain the PID, process name, memory statistics (get_mm_rss(), etc.); and obtain Cgroup information through related functions such as task_cgroup(). Step 302, System Snapshot: Directly call kernel functions, such as si_meminfo(), global_node_page_state(), etc. to obtain the global memory state, avoiding reading the / proc file system to reduce overhead; Step 4, Forced Persistence: After serializing the collected data, use the kernel_write() function and open and write it to a pre-created file with the O_DIRECT and O_SYNC flags (bypassing the page cache and writing synchronously); this operation is performed in the current process context (i.e., the OOM Killer context) and has high priority; Step 5, the process continues: After the write operation is completed (whether successful or not), the oom_notifier_call function returns NOTIFY_OK, and the OOM Killer then continues its normal process termination operation.

[0019] Example 2: Implementation via kernel patching This embodiment achieves better performance by directly modifying the Linux kernel source code (patching). Modify the kernel source code: Directly modify the out_of_memory() function in the kernel source file mm / oom_kill.c; Insert interception and acquisition logic: In the out_of_memory() function, before calling the select_bad_process() function, insert the logic of this invention; design a function oom_blackbox_snapshot(), in which information acquisition is completed; Asynchronous persistence optimization: In order to minimize the impact on OOM response latency, the collected data can be temporarily stored in a pre-allocated cache, and then a high-priority daemon kernel thread (kthread) is woken up to be responsible for the actual persistent I / O operation; in this way, the main OOM process can continue after the data is stored in the cache, without waiting for slow I / O to complete. Configure persistent regions: Specify the block device or file used to store the "black box" data in the system startup parameters or through the sysfs interface.

[0020] This invention addresses the problems of volatile fault scene information, difficulties in post-event root cause analysis, and lack of reliable "black box" records in existing Linux kernel OOM (Out of Memory) handling processes. By implementing an integrated mechanism of interception, data collection, and forced persistence at the kernel level, it achieves complete and reliable preservation of the system state at the moment of an OOM event. This invention significantly improves the maintainability and reliability of the operating system in memory exhaustion scenarios, provides a powerful basis for fault diagnosis and optimization of business systems, and has significant industrial application value.

Claims

1. A method for collecting, snapshotting, and persisting data in a Linux kernel OOM system, characterized in that, Includes the following steps: Step (1) Triggering and intercepting; Step (2) Information collection; Step (3) Force persistence; Step (4) continues.

2. The method for collecting, snapshotting, and persisting Linux kernel OOM system data according to claim 1, characterized in that, The triggering and interception described in step (1) are as follows: when the Linux kernel decides to call OOM Killer, it intercepts its default process termination process.

3. The method for collecting, snapshotting, and persisting Linux kernel OOM system data according to claim 2, characterized in that, The detailed information of the intercepted process includes: process identifier (PID), process name, OOM weight score, virtual and physical memory usage details, page table size, and the control group (Cgroup) information to which the process belongs.

4. The method for collecting, snapshotting, and persisting Linux kernel OOM system data according to claim 1, characterized in that, The information collection mentioned in step (2) specifically involves: collecting detailed information about the process selected by the OOM Killer while in the interception state, and simultaneously collecting a snapshot of the system's global memory state.

5. The method for collecting, snapshotting, and persisting Linux kernel OOM system data according to claim 4, characterized in that, The snapshot of the global memory state includes: node memory information Mem-Info, Slab cache usage, and swap space status.

6. The method for collecting, snapshotting, and persisting Linux kernel OOM system data according to claim 1, characterized in that, The forced persistence mentioned in step (3) specifically involves writing the collected process information and system state snapshots into a pre-configured persistent storage area. The persistent storage area is a non-volatile memory (NVM), a protected disk partition, or a specially marked file.

7. The method for collecting, snapshotting, and persisting Linux kernel OOM system data according to claim 1, characterized in that, The process described in step (4) continues as follows: after confirming that the information has been successfully persisted, the interception is lifted and the process termination operation of OOMKiller is continued.

8. A system for implementing the method of claims 1-7, characterized in that, include: Intercept module: Used to intercept the default process when the OOM Killer is triggered; Information acquisition module: used to collect detailed information about the selected process and a snapshot of the system's global memory state; Persistence module: Used to force the collected information to be written to the persistent storage area.

9. An apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images