A single-node fault operation and maintenance method and system based on multi-algorithm fusion

Abstract

This application provides a single-node fault operation and maintenance method and system based on multi-algorithm fusion. The method includes: collecting time-series indicators of a single-node Linux operating system; performing anomaly detection on the time-series indicators using multiple sets of anomaly detection algorithms; performing fusion processing on the output results of the multiple sets of anomaly detection algorithms to generate anomaly events; collecting kernel-state process data within the time window corresponding to the anomaly event through multiple eBPF subroutines based on graph construction logic decomposition to construct a kernel-state causal dependency graph; completing root cause localization of the anomaly event based on the causal dependency graph; determining a fault elimination strategy based on the root cause localization; and processing the anomaly event according to the elimination strategy. This invention can improve anomaly detection coverage and root cause localization accuracy, and efficiently realize single-node fault operation and maintenance.

Description

Technical Field

[0001] This invention belongs to the field of computer fault operation and maintenance technology, and in particular relates to a single-node fault operation and maintenance method and system based on multi-algorithm fusion. Background Technology

[0002] With the widespread application of cloud computing and distributed systems, the deployment scale of single-node Linux operating systems in enterprise IT environments continues to expand. The dimensions and scale of OS (Operating System) level metrics data are constantly increasing. Traditional operation and maintenance methods that rely on manual inspection and fixed threshold alarms suffer from high false alarm rates, many missed alarms, and difficulties in root cause localization. Existing intelligent operation and maintenance frameworks for Linux operating systems lack the adaptability to multiple scenarios in the anomaly detection stage. The algorithms struggle to handle single-node operating system fault detection for local density anomalies and high-dimensional metric anomalies, making it difficult to achieve high reliability and low latency kernel-level root cause localization while ensuring comprehensive anomaly detection. Summary of the Invention

[0003] This application provides a single-node fault operation and maintenance method and system based on multi-algorithm fusion. By improving the comprehensiveness of anomaly detection through the fusion of multiple anomaly detection algorithms, the root cause localization of anomaly events is completed by constructing a kernel-state causal dependency graph, which can improve the anomaly detection coverage and the accuracy of root cause localization, and efficiently realize single-node fault operation and maintenance.

[0004] This application discloses a single-node fault operation and maintenance method based on multi-algorithm fusion, including: Time-series metrics of a single-node Linux operating system are collected, and anomalies are detected by multiple sets of anomaly detection algorithms. The output results of the multiple sets of anomaly detection algorithms are fused to generate anomaly events. Kernel-mode process data within the time window corresponding to abnormal events are collected by multiple eBPF subroutines based on graph construction logic, and a kernel-mode causal dependency graph is constructed. The root cause of the abnormal event is located based on the causal dependency graph, a troubleshooting strategy is determined based on the root cause location, and the abnormal event is processed according to the troubleshooting strategy.

[0005] Optionally, constructing the kernel-mode causal dependency graph includes: The graph construction logic is broken down into multiple independent eBPF subroutines. Each eBPF subroutine performs a single atomic operation and passes the compliance check by the kernel verifier. Create a BPF program array mapping and store multiple eBPF subroutines into the corresponding index positions of the BPF program array mapping; The graph is constructed by chaining multiple eBPF subroutines in a preset order through eBPF tail calls.

[0006] Optionally, the graph construction logic can be broken down into multiple independent eBPF subroutines, including: The process metadata collection subroutine, lifecycle causal edge calculation subroutine, resource dependency causal edge calculation subroutine, IPC communication causal edge calculation subroutine, and graph adjacency matrix update subroutine are separated.

[0007] Optionally, the graph construction can be completed by chaining multiple eBPF subroutines in a preset order via eBPF tail calls, including: After the process metadata collection subroutine completes the collection and writing of process metadata, it jumps to the lifecycle causal edge calculation subroutine through the first eBPF tail call. After the lifecycle causal edge calculation subroutine completes the calculation and writing of the lifecycle causal edge, it jumps to the resource dependency causal edge calculation subroutine through the second eBPF tail call. After the resource-dependent causal edge calculation subroutine completes the calculation and writing of the resource-dependent causal edge, it jumps to the IPC communication causal edge calculation subroutine through the third eBPF tail call. After the IPC communication causal edge calculation subroutine completes the calculation and writing of the IPC communication causal edge, it jumps to the graph adjacency matrix update subroutine through the fourth eBPF tail call. The graph adjacency matrix update subroutine performs a merge update of the global causal graph data to complete the graph construction.

[0008] Optionally, the process of constructing the kernel-mode causal dependency graph also includes: Each eBPF subroutine is configured with an independent eBPF hash map as a sharded storage unit, and each sharded storage unit stores only the causal edge data generated by the corresponding eBPF subroutine. The complete causal graph data, merged by the graph adjacency matrix update subroutine, is stored through a global adjacency matrix mapping.

[0009] Optionally, the process of constructing the kernel-level causal dependency graph also includes the step of updating the causal dependency graph through an event-driven incremental update mechanism: When the kernel captures a preset trigger event, it triggers the execution of the eBPF subroutine corresponding to the preset trigger event; Only the causal edge data in the corresponding shard storage unit of the eBPF subroutine is updated, and the subroutine updated through the graph adjacency matrix only merges the changed causal edge data into the global adjacency matrix mapping. The preset triggering events include process lifecycle events, resource operation events and IPC communication events.

[0010] Optionally, the process of constructing the kernel-mode causal dependency graph also includes: The consistency of shared data is ensured by executing sequential locks and atomic operations during the serial execution of multiple eBPF subroutines in the tail call.

[0011] Optional, also includes: Cross-kernel version adaptation is achieved through cross-platform compilation based on kernel BTF type information; The abnormal events are input into the large model enhancement layer, and combined with the root cause localization results, natural language explanations and handling suggestions are generated.

[0012] Optionally, determining a troubleshooting strategy based on the root cause localization and processing the abnormal event according to the troubleshooting strategy includes: Based on the root cause localization results, obtain the identification information and abnormal operation type of the root cause process; The identification information of the root cause process and the abnormal operation type are input into the strategy generation module. The strategy generation module queries the preset operation and maintenance knowledge base, matches the handling rules corresponding to the abnormal operation type, and generates a troubleshooting strategy. The troubleshooting strategy includes operation instructions for the root cause process, recovery instructions for the affected process, and adjustment instructions for system resources. According to the operation instructions in the troubleshooting strategy, terminate or limit the root cause process, restore or restart the affected process, and adjust or isolate the system resources.

[0013] This application also discloses a single-node fault intelligent operation and maintenance system based on multi-algorithm fusion, including: The data acquisition module is used to collect time-series indicators of a single-node Linux operating system, perform anomaly detection on the time-series indicators using multiple anomaly detection algorithms, and perform fusion processing on the output results of the multiple anomaly detection algorithms to generate anomaly events. The kernel-mode tracing module is used to collect kernel-mode process data within the time window corresponding to abnormal events through multiple eBPF subroutines based on graph construction logic, and to construct a kernel-mode causal dependency graph. The root cause localization module is used to locate the root cause of an abnormal event based on the causal dependency graph, determine a troubleshooting strategy based on the root cause localization, and process the abnormal event according to the troubleshooting strategy.

[0014] As can be seen from the above scheme, this application provides a single-node fault operation and maintenance method and system based on multi-algorithm fusion. In the anomaly detection stage, the abnormal event generated by the fusion processing of the output results of multiple anomaly detection algorithms is used as the trigger condition for kernel-mode root cause localization. When an abnormal event is generated, kernel-mode process data within the corresponding time window is collected through the eBPF program. At this time, since the single program logic for directly constructing a complete causal graph exceeds the hard limit of the kernel verifier in terms of instruction count, stack depth, and loop complexity, the program loading will fail. However, this application decomposes the complete graph construction logic into multiple independent eBPF subroutines, so that each subroutine performs only a single atomic operation. Its instruction count and stack depth can be strictly controlled within a range far below the kernel verifier limit. Each subroutine can independently complete compliance verification, thereby ensuring that the loading success rate of each eBPF subroutine reaches the expected standard, breaking through the instruction count limit of a single eBPF program, and finally realizing the entire process logic of graph construction under the premise of complying with kernel verification rules, thus completing the construction of the kernel-mode causal dependency graph. The root cause localization based on this reliably constructed causal dependency graph ensures the integrity and real-time performance of the graph data because the graph data is collected and constructed in real time in kernel mode, and the execution of each subroutine is not limited by the number of instructions or stack depth of a single program. This enables the root cause localization process to reconstruct the abnormal propagation path based on the complete kernel-mode causal link, improves the accuracy of the root cause localization results, and ensures the timely and accurate handling of abnormal events. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating a single-node fault operation and maintenance method based on multi-algorithm fusion in an embodiment of this application. Figure 2 This is a flowchart illustrating a single-node fault operation and maintenance method S200 based on multi-algorithm fusion in an embodiment of this application. Figure 3 This is a flowchart illustrating S230, a single-node fault operation and maintenance method based on multi-algorithm fusion, in an embodiment of this application. Figure 4 This is a flowchart illustrating a single-node fault operation and maintenance method S400 based on multi-algorithm fusion in an embodiment of this application. Figure 5This is a flowchart illustrating a single-node fault operation and maintenance method S500 based on multi-algorithm fusion in an embodiment of this application. Figure 6 This is a flowchart illustrating a single-node fault operation and maintenance method S300 based on multi-algorithm fusion in an embodiment of this application. Figure 7 This is a schematic diagram of the structure of a single-node fault operation and maintenance system based on multi-algorithm fusion in an embodiment of this application. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not limiting, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without such specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0018] In this application embodiment, the prior art will be described in detail first. Existing intelligent operation and maintenance frameworks for Linux operating systems typically adopt a layered architecture of a collection layer, a storage layer, an analysis layer, and a presentation layer. Taking a single-node Linux server running an online business system as an example, this server is equipped with an existing intelligent operation and maintenance framework. During system operation, the collection component collects time-series indicators such as CPU utilization, memory utilization, disk I / O throughput, and network bandwidth usage at fixed intervals and stores them in a time-series database. The analysis layer reads these indicator data from the time-series database and uses a built-in single threshold detection algorithm to determine anomalies. When the number of file handles in the server's Web service process increases slowly and continuously, because the number of file handles in this process only increases by a few in each collection cycle, the overall file handle utilization of the system remains within the normal range for a considerable period. The single threshold detection algorithm, because the detection threshold is set within the normal range of the overall system, fails to identify the abnormal change in the local file handle density of this process, resulting in a false negative. When the server experiences irregular and large fluctuations in multiple metrics such as CPU utilization, memory usage, and disk I / O rate due to abnormal background task triggering, the single threshold detection algorithm can only make independent threshold judgments for each metric. It cannot comprehensively utilize the correlation between multi-dimensional metrics to identify such high-dimensional abnormal patterns, resulting in insufficient detection sensitivity and false alarms.

[0019] Based on this, embodiments of this application provide a single-node fault operation and maintenance method based on multi-algorithm fusion, such as... Figure 1 As shown, it includes: S100: Collect time-series metrics of a single-node Linux operating system, perform anomaly detection on the time-series metrics using multiple anomaly detection algorithms, perform fusion processing on the output results of the multiple anomaly detection algorithms, and generate anomaly events; S200: Collect kernel-mode process data within the time window corresponding to abnormal events through multiple eBPF subroutines based on graph-based construction logic, and construct a kernel-mode causal dependency graph; S300: Based on the causal dependency graph, complete the root cause localization of the abnormal event, determine the troubleshooting strategy based on the root cause localization, and process the abnormal event according to the troubleshooting strategy.

[0020] The abnormal events generated by fusing the outputs of multiple anomaly detection algorithms are used as trigger conditions for kernel-mode root cause localization. After an abnormal event is generated, kernel-mode process data within the corresponding time window is collected via the eBPF program. At this point, because the single-program logic for directly constructing the complete causal graph exceeds the hard limits of the kernel verifier in terms of instruction count, stack depth, and loop complexity, program loading will fail. This application decomposes the complete graph construction logic into multiple independent eBPF subroutines, ensuring that each subroutine performs only a single atomic operation. Its instruction count and stack depth can be strictly controlled to be far below the kernel verifier's limits. Each subroutine can independently complete compliance verification, thereby ensuring that the loading success rate of each eBPF subroutine reaches the expected standard, breaking through the instruction count limit of a single eBPF program. Finally, under the premise of complying with kernel verification rules, the entire process logic for graph construction is fully implemented, completing the construction of the kernel-mode causal dependency graph. The root cause localization based on this reliably constructed causal dependency graph ensures the integrity and real-time performance of the graph data because the graph data is collected and constructed in real time in kernel mode, and the execution of each subroutine is not limited by the number of instructions or stack depth of a single program. This enables the root cause localization process to reconstruct the abnormal propagation path based on the complete kernel-mode causal link, improves the accuracy of the root cause localization results, and ensures the timely and accurate handling of abnormal events.

[0021] It should be noted that time-series metrics refer to a set of operating system runtime status data that changes continuously over system runtime and is recorded in chronological order of occurrence. These data can reflect the system's runtime trends and instantaneous state over time. Time-series metrics may include various data directly related to the operating system's runtime status, such as CPU utilization, memory utilization, disk I / O throughput, network bandwidth usage, process runtime status, system call frequency, context switch count, number of processes, and number of file handles used.

[0022] The data collection operation can be completed by a data collection component deployed on the target single node. The data collection component can periodically read the relevant virtual file system of the Linux operating system or call the relevant data interface provided by the operating system to obtain the current values ​​of the above indicators. The collected time-series indicators can be stored in the time-series database in the order of collection time for subsequent reading and analysis.

[0023] It should be noted that after collecting time-series indicators, this method performs anomaly detection on the collected indicators using multiple sets of anomaly detection algorithms. Multiple sets of anomaly detection algorithms refer to the simultaneous running of two or more anomaly detection algorithms with different detection principles. Each algorithm judges the same batch of time-series indicators from its own different dimensions and outputs its own anomaly detection results. Different types of anomaly detection algorithms complement each other in their detection principles, and the combined use of multiple algorithms can cover a wider range of anomaly scenarios, compensating for the limitations of a single algorithm in detecting specific anomaly types. For example, the multiple sets of anomaly detection algorithms may include the Isolation Forest algorithm and the Local Anomaly Factor algorithm. Furthermore, the multiple sets of anomaly detection algorithms may also include the Long Short-Term Memory Network anomaly detection algorithm. In this application, the selection and combination of anomaly detection algorithms are managed through configuration files, allowing operations and maintenance personnel to flexibly configure them according to the business characteristics and historical fault types of the nodes.

[0024] In this embodiment, after anomaly detection by multiple anomaly detection algorithms is completed, the anomaly detection results output by each algorithm are fused to generate the final anomaly event. The output results of the multiple anomaly detection algorithms include an anomaly score or anomaly label output by each algorithm for the current time-series index data. The anomaly score is a continuous value, such as a value between 0 and 1. The higher the value, the greater the probability that the data point is anomaly, and the lower the value, the greater the probability that the data point is normal. The anomaly label can be a discrete Boolean value, indicating whether the data point is directly determined to be anomaly.

[0025] Fusion processing can employ a voting fusion method, where the final anomaly label is determined by majority voting. When more than a preset number of anomaly detection algorithms simultaneously classify the current sample as anomaly (e.g., two or more out of three algorithms classify it as anomaly), the fused result is deemed anomaly, generating a corresponding anomaly event. Alternatively, a weighted fusion method can be used. This involves assigning a preset weight coefficient to each anomaly detection algorithm based on its historical performance. The anomaly scores output by multiple algorithms are then weighted and averaged. The resulting composite anomaly score is compared to a preset fusion threshold. If the composite score exceeds this threshold, the fused result is deemed anomaly, generating a corresponding anomaly event. The weight coefficients can be set based on the detection performance of each algorithm in different anomaly scenarios. Algorithms with better detection performance are assigned higher weight coefficients, while those with average performance are assigned lower weight coefficients, with the sum of all weight coefficients being 1.

[0026] In alternative implementations, such as Figure 2 As shown, the S200 constructs a kernel-mode causal dependency graph including: S210: Decompose the graph construction logic into multiple independent eBPF subroutines. Each eBPF subroutine performs only a single atomic operation and independently passes the compliance check of the kernel verifier. S220: Create a BPF program array mapping and store multiple eBPF subroutines into the corresponding index positions of the BPF program array mapping; S230: The graph is constructed by chaining multiple eBPF subroutines in a preset order through eBPF tail calls. The eBPF tail calls are a stackless execution method between eBPF programs implemented based on kernel helper functions.

[0027] In this embodiment, after an abnormal event is generated, kernel-mode process data within the corresponding time window of the abnormal event is collected using an eBPF program to construct a kernel-mode causal dependency graph. eBPF is a technology that allows user-defined programs to run securely in the Linux kernel space. It provides a secure sandbox mechanism for executing custom data processing logic in kernel space without modifying the kernel source code or loading kernel modules.

[0028] An abnormal event's corresponding time window refers to a continuous time interval of a preset length, starting from the specific time the abnormal event occurred and going back to an earlier time. For example, if an abnormal event is generated at time T0, and the preset backtracking time is 30 seconds, then the time window corresponding to the abnormal event is the 30-second time interval between time T0 minus 30 seconds and time T0.

[0029] Kernel-level process data includes kernel-level data directly related to the running state of processes, such as process lifecycle information, parent-child relationships, resource holding and waiting information, inter-process communication information, and system call execution information. Process lifecycle information includes process creation time, destruction time, process identifier (PID), parent process identifier (PPID), and namespace identifier, reflecting the complete lifecycle trajectory of the process from creation to destruction. Resource holding and waiting information includes the number of memory pages currently held by the process, the list of open file descriptors, the type and status of held kernel locks, the type of resource being waited for, and the waiting time, reflecting the process's occupation and competition for system resources. Inter-process communication information includes pipe read / write events, Unix domain socket message sending and receiving events, and shared memory attachment and detachment events, reflecting the data interaction between processes through inter-process communication mechanisms. This kernel-level data comprehensively records the causal dependencies between processes from multiple dimensions, serving as the fundamental data source for constructing causal dependency graphs.

[0030] A kernel-mode causal dependency graph is a directed graph structure composed of process nodes and causal edges. Process nodes correspond to currently running or previously running processes in the operating system. Each process node is uniquely identified by its process identifier (PID) and includes its metadata. A causal edge is a directed line segment pointing from a source process node to a target process node, indicating that the source process has a causal impact on the target process in some way. The weight of a causal edge is a quantified value representing the strength of the causal dependency; a larger weight indicates a stronger causal relationship between the two processes. Through the organic combination of process nodes and causal edges, the causal dependency graph can reconstruct the complete causal propagation chain from the root cause process to the process exhibiting the anomaly during an anomaly. For example, if process A creates process B, there is a lifecycle causal edge in the graph pointing from A to B; if process B is blocked waiting for a lock resource held by process A, there is a resource dependency causal edge in the graph pointing from A to B; if process B sends data to process C through a pipe, causing C to start processing a task, there is an IPC communication causal edge in the graph pointing from B to C. By traversing the propagation path formed by these causal edges, one can trace back from the anomalous behavior node to the original root cause node.

[0031] It's important to note that constructing the kernel-mode causal dependency graph first decomposes the complete graph construction logic into multiple independent eBPF subroutines. Each eBPF subroutine performs only a single atomic operation. A single atomic operation means that the subroutine is responsible for performing only one independent, indivisible task throughout the entire graph construction process. For example, one subroutine might be responsible for collecting process metadata from kernel structures, another for calculating parent-child causal edges between processes based on process lifecycle events, another for calculating resource dependency causal edges between processes based on resource wait events, and yet another for merging and updating the causal edge data from each slice into the global graph data. After decomposition, the code logic of each subroutine is simple and clear, without complex branching or nested loops. The total number of instructions can be capped at 500, and the stack space depth used during program execution can be capped at 128 bytes.

[0032] The Linux kernel eBPF verifier imposes explicit hard limits on the number of instructions, stack depth, and loop rules for loading a single eBPF program. The maximum number of instructions per program is generally one million, but complex programs often reach this limit earlier due to other constraints. The hard limit for stack depth is 512 bytes, and loops must have a statically determinable upper bound. Each small subroutine, due to its singular function and simple logic, meets all the verifier's hard limits on instruction count and stack depth. Each subroutine can independently pass the kernel verifier's compliance check, with a deterministic pass rate. However, if existing technologies are used to centralize all functions—including process metadata collection, lifecycle causal edge calculation, resource dependency causal edge calculation, IPC communication causal edge calculation, and graph adjacency matrix updates—into a single eBPF program, the code logic of this monolithic program becomes complex. The overlapping instructions of various functional modules cause the total number of instructions to exceed the verifier's acceptable range. Multiple nested function calls cause the runtime stack depth to exceed the 512-byte limit, and some loop counts depend on the runtime state, making static determination impossible during the verification period. These factors collectively lead to a higher probability that the monolithic program will be intercepted by the verifier and fail to load. This application decomposes the complete graph construction logic into multiple independent eBPF subroutines, each of which performs only a single atomic operation. This avoids the verification failure problem caused by the overly concentrated and complex logic of a single program from the program architecture level, ensuring that the verification and loading of each subroutine are deterministic.

[0033] S210 decomposes the map construction logic into multiple independent eBPF subroutines, including: S211: This subroutine is divided into the following subroutines: process metadata collection, lifecycle causal edge calculation, resource dependency causal edge calculation, IPC communication causal edge calculation, and graph adjacency matrix update.

[0034] The graph construction logic is broken down into five subroutines: program process metadata collection subroutine, lifecycle causal edge computation subroutine, resource dependency causal edge computation subroutine, IPC communication causal edge computation subroutine, and graph adjacency matrix update subroutine. Each of these five subroutines independently undertakes a single atomic operation in one stage of the graph construction process. They are executed in series in a preset order through the eBPF tail call mechanism, and work together to complete the entire process from raw kernel data collection to final causal graph generation.

[0035] The process metadata collection subroutine is an eBPF program of type eBPF_PROG_TYPE_TRACEPOINT, which can be attached to the tracepoint hooks sched_process_fork and sched_process_exit in the Linux kernel. When a process creation event occurs in the kernel, i.e., when a parent process creates a child process through a fork or clone system call, the kernel triggers the sched_process_fork tracepoint. At this time, the process metadata collection subroutine attached to this hook is automatically called and executed by the kernel. This subroutine extracts core metadata information related to the newly created process from the kernel task_struct structure of the current process, including the host PID assigned to the new process, the PID of its parent process, the namespace ID to which the process belongs, the process's start timestamp, and the process's current running state, such as ready, running, or blocked. The extracted process metadata structure containing the above information is written into an eBPF global hash map proc_meta_map, with the host PID as the key and the process metadata structure as the value.

[0036] When a process destruction event occurs in the kernel, i.e., when a process calls the exit system call to exit or is forcibly terminated by the kernel, the kernel will trigger the sched_process_exit tracepoint. At this time, the process metadata collection subroutine attached to this hook is called and executed again. This subroutine looks up the entry corresponding to the PID of the destroyed process in proc_meta_map, deletes it from the map and cleans it up, ensuring that proc_meta_map only stores the latest metadata of the currently active processes in the system at any time, and does not leave behind historical expired data of processes that have already exited.

[0037] The lifecycle causal edge computation subroutine is also an eBPF program of type eBPF_PROG_TYPE_TRACEPOINT, which can be attached to the kernel's sched_process_exec tracepoint hook. When a process exec event occurs in the kernel, that is, when a process loads and executes a new executable file through the execve system call, replacing the original code and data of the current process with the code and data of the new program, the kernel triggers the sched_process_exec tracepoint, and the lifecycle causal edge computation subroutine is called and executed. This subroutine reads the metadata of the process currently performing the exec operation from proc_meta_map, obtains relevant information about the parent process based on its parent PID, and constructs a lifecycle causal edge from the parent process to the current child process based on the parent-child relationship and the exec execution chain. This causal edge represents the lifecycle dependency relationship between the parent process and the child process created by the parent process and its execution. This subroutine calculates the weight of the lifecycle causal edge. The weight can be set according to the tightness of the relationship between the parent and child processes. For example, the weight of a causal edge with a direct parent-child process relationship can be set to a higher value, while the weight of a causal edge with a relationship formed indirectly through an intermediate process decreases accordingly. After the calculation is completed, the constructed causal edge and its weight are written into the corresponding eBPF hash map, and the entry is marked as an incremental update entry for use in subsequent incremental merge operations.

[0038] The resource dependency causal edge calculation subroutine is an eBPF program of type eBPF_PROG_TYPE_KPROBE, which can be attached to the kprobe hooks of kernel functions related to resource operations, such as sys_mmap, sys_munmap, and filp_close. By attaching to kernel functions such as memory mapping, memory release, and file closing, this subroutine can capture the holding, waiting, and releasing behaviors of each process for system resources in real time. Internally, this subroutine contains strict timing and resource verification logic. A resource dependency causal edge is constructed only when both of the following conditions are met: Condition 1 is that process B is currently in a blocked state waiting for a certain resource; Condition 2 is that the resource being waited for is currently exclusively held by process A and has not yet been released. Only when both of these conditions are met is a resource dependency causal edge from process A to process B constructed. This causal edge represents the resource dependency relationship where process A's exclusive holding of the resource causes process B to be blocked while waiting for that resource. This subroutine calculates the weights of causal edges based on the resource type and the duration of the wait. Higher base weights are assigned to highly exclusive resource types such as mutexes, while lower base weights are assigned to shareable resource types such as read locks. Longer wait times correspond to higher weights. After calculation, the constructed causal edges and their weights are written into the corresponding eBPF hash map and marked as incremental update entries.

[0039] The IPC communication causal edge calculation subroutine is an eBPF program of type eBPF_PROG_TYPE_TRACEPOINT, which can be attached to kernel tracepoint hooks related to inter-process communication, such as pipe_read, pipe_write, and unix_stream_sendmsg. When IPC communication events such as pipe reading, pipe writing, and Unix domain socket message sending occur in the kernel, the corresponding tracepoint is triggered, and this subroutine is called and executed. This subroutine captures information such as the PIDs of the sending and receiving processes of the IPC message, the direction of communication, and the size of the message data, and strictly verifies the validity of the timing of the event. Only when the message sending time is earlier than the time the message is received by the receiver (i.e., the timestamp of the sending event is less than the timestamp of the receiving event) is it confirmed as a valid IPC communication causal chain, constructing an IPC communication causal edge from the message sending process to the message receiving process. This causal edge indicates that the sending process transmits data or signals to the receiving process through the inter-process communication mechanism, thereby triggering or affecting the behavior of the receiving process. This subroutine calculates the weights of causal edges based on the amount of communication data and the frequency of communication. The larger the data volume or the higher the communication frequency, the greater the weight of the causal edge. After calculation, the constructed causal edges and their weights are written into the corresponding eBPF hash map and marked as incremental update entries.

[0040] The graph adjacency matrix update subroutine is an eBPF program of type eBPF_PROG_TYPE_RAW_TRACEPOINT. Its execution is not directly triggered by kernel events, but rather by the preceding causal edge calculation subroutine via the eBPF tail call mechanism. The core execution logic of this subroutine is to traverse and read the causal edge records marked as incremental update entries in each dimension's sharding map, and then merge these newly added or changed causal edge data one by one into the global sparse causal graph adjacency matrix map. If the causal edge does not yet exist in the global adjacency matrix, it is added directly; if it already exists, the newly calculated weight overwrites or updates the original weight. After the merge and update are complete, this subroutine is also responsible for cleaning up expired causal edge data. For example, when a process has exited and been destroyed, all causal edges related to that process will be removed from the global adjacency matrix, ensuring that the causal graph data stored in the global adjacency matrix map always reflects the latest inter-process causal relationship state of the current system, and that no historically expired data remains to affect the accuracy of root cause localization.

[0041] In alternative implementations, such as Figure 3 As shown, S230 completes the map construction by chaining multiple eBPF subroutines in a preset order through eBPF tail calls, including: S231: After the process metadata collection subroutine completes the collection and writing of process metadata, it jumps to the lifecycle causal edge calculation subroutine through the first eBPF tail call; S232: After the lifecycle causal edge calculation subroutine completes the calculation and writing of the lifecycle causal edge, it jumps to the resource dependency causal edge calculation subroutine through the second eBPF tail call; S233: After the resource-dependent causal edge calculation subroutine completes the calculation and writing of the resource-dependent causal edge, it jumps to the IPC communication causal edge calculation subroutine through the third eBPF tail call; S234: After the IPC communication causal edge calculation subroutine completes the calculation and writing of the IPC communication causal edge, it jumps to the graph adjacency matrix update subroutine through the fourth eBPF tail call; S235: The graph adjacency matrix update subroutine performs a merge update of the global causal graph data.

[0042] In this embodiment, after splitting the aforementioned multiple eBPF subroutines, a BPF program array map is created, and the multiple eBPF subroutines are stored in the corresponding index positions of the BPF program array map. The type of the BPF program array map is eBPF_MAP_TYPE_PROG_ARRAY, which is a special type of kernel mapping structure designed by the Linux kernel mainly to support the eBPF tail call mechanism. Its purpose is to store a set of pointer references to a group of eBPF programs to be executed. The key of the map is a 32-bit unsigned integer, representing the sequential index position of the subroutine in the map, with the index value starting from 0 and increasing sequentially; the value of the map is the kernel program pointer corresponding to the file descriptor of the eBPF program stored at the corresponding index position.

[0043] When creating a BPF program array map, the maximum entry capacity of the map needs to be preset. This maximum capacity must be greater than the total number of subroutines that actually need to be stored, typically twice or more than the actual number of subroutines. Sufficient redundant index positions are reserved for future functional expansion. For example, if a new subroutine is added later to handle network event causal edges, it can be directly stored in a reserved index position without modifying the existing program structure. After creating the BPF program array map, each subroutine is stored in its corresponding index position according to the preset index allocation scheme: the process metadata collection subroutine is stored in index 0, serving as the entry point for the entire tail call execution chain; the lifecycle causal edge calculation subroutine is stored in index 1; the resource dependency causal edge calculation subroutine is stored in index 2; the IPC communication causal edge calculation subroutine is stored in index 3; the graph adjacency matrix update subroutine is stored in index 4; and the exception fallback subroutine is stored in index 5 as a fallback handling procedure for exceptions. Each subroutine has a one-to-one correspondence with its index position, so there will be no index conflict that could lead to calling the wrong target program.

[0044] For example, multiple eBPF subroutines can be chained together in a preset index order via eBPF tail calls to complete the construction of the entire flowchart. eBPF tail calls are a seamless inter-eBPF program jump execution method implemented based on the kernel helper function `eBPF_tail_call`. The function signature of this kernel helper function is `long eBPF_tail_call(void*ctx, struct eBPF_map*map, u32index)`, where the `ctx` parameter is a pointer to the context information of the current eBPF program. During the tail call jump, this context is passed intact to the target subroutine to ensure the consistency of the execution context. The `map` parameter is a pointer to a BPF program array mapping used to find the target jump program, and the `index` parameter is the index value of the target subroutine in the program array mapping used to specify the jump target.

[0045] When the eBPF_tail_call function executes successfully, the calling program will not continue executing any subsequent code. The execution flow jumps directly to the first instruction of the target subroutine. The calling program's stack frame is not preserved, so the calling program's stack space is completely released and does not accumulate stack depth. Because the caller's stack frame is not preserved during the entire jump process, regardless of how many tail call jumps occur during the entire graph construction process, from the first call to the last call, the stack depth during program execution is always equal to the stack depth of the currently executing subroutine itself. This prevents the stack depth from accumulating and exceeding the kernel's 512-byte hard limit due to a large number of subroutines.

[0046] When performing compliance checks on each eBPF subroutine, the kernel eBPF verifier employs an independent verification strategy. It analyzes only the instruction count, stack depth, and loop logic of each subroutine individually, without merging the instruction count of the called program into the total instruction count of the calling program for verification. This independent verification mechanism, combined with the stackless jump characteristic of tail calls, allows the entire graph construction scheme to construct a complete and complex graph construction logic far exceeding the instruction limit of a single program, provided that each subroutine passes verification. Under the premise of complying with all kernel verification rules, it fully realizes the end-to-end graph construction function from process metadata collection, causal edge calculation in various dimensions to global adjacency matrix update.

[0047] Before executing a jump, the kernel helper function `eBPF_tail_call` performs three validity checks. The actual jump only occurs if all three checks pass. The first check is an index out-of-bounds check, which examines whether the passed `index` value is less than the maximum entry capacity preset in the BPF program array map. If `index` is greater than or equal to the maximum entry capacity, the check fails, `eBPF_tail_call` returns a negative error code, and continues execution of the calling program's subsequent code. The second check is a maximum nesting level check. Each CPU core maintains a tail call nesting depth counter, which increments each time a tail call jump is executed. When the counter reaches or exceeds the kernel-specified maximum tail call nesting level of 33, the check fails, and `eBPF_tail_call` returns an error. The third check is a target program existence check, which checks whether the entry position corresponding to the `index` in the BPF program array map already stores a valid `eBPF` program pointer. If the index position is empty and no valid program reference is stored, the check fails, and `eBPF_tail_call` returns an error. The above triple verification mechanism together ensures the security and reliability of tail call jump.

[0048] Specifically, the execution flow of multiple eBPF subroutines, sequentially linked according to their indexes, is as follows: The process metadata collection subroutine serves as the entry point for the entire tail call execution chain. When a process event occurring in the kernel triggers the tracepoint hook attached to this subroutine, it is first called and executed by the kernel to complete its responsible operations of collecting process metadata and writing it to the eBPF hash map. After completing all its collection and writing logic, this subroutine calls the kernel helper function `eBPF_tail_call`, passing in the current program execution context, the file descriptor of the BPF program array mapping, and index value 1, requesting a jump to the lifecycle causal edge calculation subroutine corresponding to index 1. Once activated by the jump, the lifecycle causal edge calculation subroutine begins execution, reading process metadata from the relevant eBPF hash map, completing the calculation and writing of lifecycle causal edges, and after completion, calling the `eBPF_tail_call` function with index value 2 to jump to the resource dependency causal edge calculation subroutine. After the resource-dependent causal edge calculation subroutine completes the calculation and writing of resource-dependent causal edges, it calls `eBPF_tail_call` with index value 3 and jumps to the IPC communication causal edge calculation subroutine. After completing the calculation and writing of IPC communication causal edges, the IPC communication causal edge calculation subroutine calls `eBPF_tail_call` with index value 4 and jumps to the graph adjacency matrix update subroutine. The graph adjacency matrix update subroutine performs a global causal graph data merging and updating operation, merging the incremental update entries marked in each partition mapping into the global adjacency matrix mapping and cleaning up expired data. After its execution, the entire graph construction process is complete, and no further tail call jumps are performed. The entire serial execution chain adopts a strictly linear, branchless jump design. Each subroutine only needs to execute one tail call to transfer the execution flow to the next subroutine. The jump logic is extremely simple and does not increase the control flow complexity of the program. In this embodiment, with the five-subroutine concatenation scheme, the maximum nested level of the tail call is only five levels, far below the kernel's maximum limit of 33 levels. All legality checks pass smoothly, ensuring a deterministic jump success rate. Furthermore, because the tail call uses a stackless jump method, throughout the entire execution process, from the first subroutine to the last, the program's stack depth remains at the level of a single subroutine's stack depth, with an upper limit controlled within 128 bytes, far below the kernel's hard stack depth limit of 512 bytes, eliminating the risk of stack overflow.

[0049] An additional fallback subroutine is stored at index 5 of the BPF program array mapping. This fallback subroutine is a backup program for handling exceptional situations. When an exception occurs during the tail call jump, such as an index out of bounds error or the target subroutine failing to load, causing the `eBPF_tail_call` function to fail and return an error code, the calling program catches the error return, executes the subsequent exception handling logic, and calls `eBPF_tail_call` with index 5 to jump to the fallback subroutine. The fallback subroutine performs data cleanup operations, such as clearing incomplete intermediate state data, resetting related state flags, and releasing allocated temporary storage space, before safely exiting. This prevents incomplete or inconsistent dirty data from remaining in the kernel data structures due to intermediate subroutine jump failures, which could lead to kernel crashes, data corruption, or memory leaks, thus improving the robustness of the entire eBPF graph construction scheme for continuous and stable operation in a production environment.

[0050] In alternative implementations, such as Figure 4 As shown, the process of constructing the kernel-mode causal dependency graph also includes S400: S410: Configure an independent eBPF hash map as a sharded storage unit for each eBPF subroutine, and each sharded storage unit stores only the causal edge data generated by the corresponding eBPF subroutine.

[0051] S420: The complete causal graph data, merged by the graph adjacency matrix update subroutine, is stored through a global adjacency matrix mapping. The eBPF hash mapping is a key-value pair storage structure accessible in both kernel and user space.

[0052] In this embodiment, during the construction of the kernel-mode causal dependency graph, an independent eBPF hash map is configured as a fragmented storage unit for each eBPF subroutine. Each fragmented storage unit stores only the causal edge data generated by the corresponding eBPF subroutine, and the data is physically isolated from each other. A global adjacency matrix mapping is used to store the complete causal graph data merged by the graph adjacency matrix update subroutines. The eBPF hash map is of type eBPF_MAP_TYPE_HASH, which is a key-value pair storage structure based on a hash table provided by the Linux kernel. This storage structure can be accessed in both kernel space and user space. Kernel-mode eBPF programs can read and write key-value pairs in the mapping through eBPF helper functions such as eBPF_map_update_elem and eBPF_map_lookup_elem, while user-mode applications can access the same mapping through eBPF system calls. Thanks to the hash table implementation mechanism, the time complexity of read and write access operations for a single key-value pair in the mapping is O(1). That is, the time consumption of the access operation does not increase with the increase of the total number of entries stored in the mapping, which can ensure stable low access latency under various data volumes. The kernel provides a built-in concurrency protection mechanism for each eBPF mapping. When multiple CPU cores perform read and write operations on the same mapping at the same time, the kernel uses synchronization mechanisms such as spinlocks to ensure that only one operation modifies the internal pointer of the mapping data structure at any given time. Users do not need to manually add locking and unlocking logic in the eBPF program code, which can safely avoid the problem of data corruption or loss caused by race conditions that may occur during multi-core concurrent access.

[0053] In alternative implementations, such as Figure 5 As shown, the process of constructing the kernel-mode causal dependency graph also includes updating the causal dependency graph S500 through an event-driven incremental update mechanism: S510: When the kernel captures a preset trigger event, it triggers the execution of the eBPF subroutine corresponding to the preset trigger event.

[0054] S520: Only update the causal edge data in the corresponding shard storage unit of the eBPF subroutine, and only merge the changed causal edge data into the global adjacency matrix mapping through the graph adjacency matrix update subroutine; the preset trigger events include process lifecycle events, resource operation events and IPC communication events.

[0055] In this embodiment, an event-driven incremental update mechanism is used throughout the graph construction process to update the causal dependency graph. Event-driven means that specific types of events occurring in the kernel trigger the process. Only when an event occurs is the corresponding subroutine executed and the incremental update of the corresponding slice performed. When no event occurs, the entire graph construction system remains in a silent waiting state and does not consume computing resources. The core logic of this incremental update mechanism is to perform local updates only on the graph slices that have changed and their corresponding adjacency matrix entries. No operations are performed on the slice data and adjacency matrix entries that have not changed. This reduces the computational cost of a single graph update from a scale related to the total number of processes and the total number of causal edges in the system to a constant scale directly related to a single event.

[0056] When the kernel captures a preset trigger event, it triggers the execution of the corresponding eBPF subroutine based on the specific type of the event. This subroutine only calculates and writes a few causal edge data in the causal edge shard storage unit it is responsible for. After the calculation is completed, it marks them as incremental update entries. Then, it triggers the execution of the graph adjacency matrix update subroutine through the eBPF tail call. This update subroutine only reads those entries with incremental update marks in the shard map, merges them to update the global adjacency matrix map, and clears the incremental update marks of these entries for future use. Large quantum entries in each shard map that do not have incremental marks are not read or processed at all.

[0057] The preset trigger events are divided into three categories based on causal edge types. The first category is process lifecycle events, specifically including process creation events, process destruction events, and process exec execution events. These events trigger the execution of the process metadata collection subroutine and the lifecycle causal edge calculation subroutine, updating the process metadata records in `proc_meta_map` and the lifecycle causal edge data in `life_cycle_slice_map`. The second category is resource operation events, specifically including memory allocation and release events, file open and close events, kernel lock holding and release events, and CPU scheduling wait events. These events trigger the execution of the resource dependency causal edge calculation subroutine, updating the resource dependency causal edge data in `resource_dep_slice_map`. The third category is IPC communication events, specifically including pipe read / write events, Unix domain socket message sending / receiving events, and shared memory attach / detach events. These events trigger the execution of the IPC communication causal edge calculation subroutine, updating the IPC communication causal edge data in `ipc_comm_slice_map`.

[0058] The complete execution flow of the incremental update mechanism consists of three steps. The first step is the event triggering and slice update step. The kernel captures a preset triggering event on a certain CPU core, such as an event where a process requests memory through sys_mmap. This event is captured by the kprobe hook attached to the resource dependency causal edge calculation subroutine. The subroutine starts executing and reads the process's metadata from proc_meta_map. Combined with the specific information of the event, such as the size and address of the requested memory, it calculates the causal edges directly related to the event. There may only be a few such edges. For example, if the process enters the waiting queue because of requesting memory, and the memory resource is being held and used by another process, then a resource dependency causal edge from the resource holding process to the current waiting process is constructed. This causal edge is written to resource_dep_slice_map and the incremental flag field is set to the pending update state. The second step is the incremental merging and global update step. After completing the writing and marking operations of its own shards, this subroutine jumps to the graph adjacency matrix update subroutine via an eBPF tail call. After the update subroutine starts executing, it iterates through all entries in the shard mappings whose incremental marker fields are in the pending update state, reads the causal edge data of these entries, merges and updates them into the global sparse adjacency matrix adj_matrix_map, updates existing edges with new weights, and directly adds new edges that do not exist. After the merging and update are completed, the incremental marker fields of these entries are cleared from the pending update state. The third step is the expired data cleanup step. When a process exits the system due to a destruction event, the process metadata collection subroutine not only deletes the process's metadata record from proc_meta_map, but also notifies the graph adjacency matrix update subroutine, which searches for and deletes all causal edge records in the global adjacency matrix adj_matrix_map that have the PID of the destroyed process as the source process or target process, releasing these no longer needed storage spaces, while leaving the causal edge data between other surviving processes completely unmodified.

[0059] Under normal daily operation, the total number of the three types of preset triggered events occurring per unit time on a single-node Linux server is typically small, such as an average of ten to several dozen times per second, far lower than the total number of active processes on the single node. Each event-triggered incremental update only needs to process a very small number of causal edge entries, usually a few to a dozen each time, keeping the graph update latency at the microsecond level. Even if the number of active processes on a single node is large, the processing latency of a single incremental update does not increase significantly with the increase in the total number of processes. In contrast, the full recalculation mode requires a global recalculation and scan of all active processes and all causal edge relationships in the system for each update. Its single update latency reaches the millisecond level or even higher when the number of processes is large, showing a clear positive correlation with the system size. Furthermore, the incremental update mechanism incurs no computational overhead when no events occur. Each eBPF subroutine is in a mounted waiting state but does not actually execute any instructions, thus not consuming CPU time slices. The CPU usage of the entire graph construction system is at a low level when running in a production environment, and it does not cause significant resource contention affecting the normal operation of business processes.

[0060] The process of constructing the kernel-mode causal dependency graph also includes S600: S610: By executing sequence locks and atomic operations, the consistency of shared data among multiple eBPF subroutines during the cascading execution of tail calls is ensured. The execution sequence lock is a global execution status flag set in a specified mapping, used to identify the execution stage of the current tail call chain.

[0061] In this embodiment, multiple eBPF subroutines are executed sequentially via tail calls. They share global eBPF hash mapping data for read and write operations, requiring appropriate concurrency control measures to ensure data consistency and correctness during read and write operations, and to avoid problems such as read / write conflicts, dirty data, and data loss caused by concurrent access or disordered execution order. This application, in constructing the kernel-mode causal dependency graph, uses a combination of execution sequence locks and atomic operations to ensure the consistency of shared data during the sequential execution of multiple subroutines via tail calls.

[0062] Execution sequence locks are global execution status markers set in `proc_meta_map` to track and identify the execution stage of the current tail call chain. Specifically, a global execution status entry with a fixed key is created in `proc_meta_map`. The value of this entry is an integer execution stage number, and the entire graph construction process executes sequentially from stage one to stage five. Initially, the execution stage number is 0, indicating that there is no tail call chain currently in progress.

[0063] When the process metadata collection subroutine is triggered by a kernel event and begins execution, it first updates the execution status entry value in `proc_meta_map` to 1 using an atomic operation, indicating that it is currently in the first stage, i.e., the process metadata collection stage. Then, it performs metadata collection and writing operations. After completion, it updates the execution status entry value to 2, indicating that the first stage is complete and it enters the second stage. Then, it calls `eBPF_tail_call` to jump to the lifecycle causal edge computation subroutine. When the lifecycle causal edge computation subroutine begins execution, it first reads the execution status entry value in `proc_meta_map` and verifies whether it equals the number 2 corresponding to the second stage. Only after the verification passes, confirming that the preceding stage is complete, does it formally begin executing its own causal edge computation logic, avoiding premature reading of incomplete data before the metadata of the preceding stage is fully written. After this subroutine completes execution, it updates the execution status entry value to 3 and jumps to the next subroutine. Each subsequent subroutine first verifies whether its execution status matches the number of its corresponding stage. Only after the verification passes does it execute, and after completion, it updates the status to the next stage number before jumping. Through this phase verification and update mechanism of execution sequence lock, it is strictly guaranteed that each subroutine is executed in the order of the preset first to fifth stages. The next stage of the program will only start executing after the previous stage has been completely completed. This completely avoids the problem of data inconsistency caused by subsequent subroutines reading intermediate state data that has not yet been written by the previous subroutine due to disordered execution order.

[0064] The write and update operations for specific data entries in the eBPF hash map are completed using atomic operation helper functions provided by the eBPF kernel. An atomic operation is an operation guaranteed by hardware at the underlying computer architecture to be uninterruptible. This operation either executes completely successfully and writes the result to the target address, or it does not execute at all and the value at the target address remains unchanged. It will never be interrupted halfway through execution, resulting in partial data writing while other data is not. When multiple eBPF subroutines on multiple CPU cores may simultaneously need to update the same key-value pair record in the same eBPF hash map due to their respective event triggers, the atomic operation mechanism ensures that at any given time, only the operation on one core can complete the update of the entry. Subsequent operations on other cores see the complete updated data, and the updated result will not be overwritten or lost. This avoids the data corruption problem that may occur when multiple cores write to the same memory location simultaneously due to race conditions. By using execution sequence locks to ensure the correct execution order of subroutines at a macro level, and by using atomic operations to ensure that single-entry write and update operations are uninterrupted at a micro level, these two mechanisms work together at different levels to guarantee data consistency throughout the entire graph construction process.

[0065] The fault operation and maintenance method also includes cross-kernel version adaptation of the eBPF subroutine through cross-platform compilation based on kernel BTF type information; The abnormal events are input into the large model enhancement layer, and combined with the root cause localization results, natural language explanations and handling suggestions are generated.

[0066] For example, based on the kernel-level causal dependency graph constructed above, the root cause of anomalies can be located. The constructed causal dependency graph is stored in a global adjacency matrix mapping in the form of a directed graph. Each node in the graph represents a process, and each directed edge represents the causal influence relationship between one process and another. The weight of the edge represents the strength of the influence. After an anomaly is generated, the graph has accumulated and recorded the complete causal propagation relationship between processes within the anomaly time window. The root cause contribution of each process node in the entire graph can be calculated using a node importance analysis algorithm based on the graph structure. During the calculation, the out-degree and in-degree of each node are considered comprehensively. The out-degree represents the breadth of the influence that process exerts on other processes, and the in-degree represents the degree to which the process is affected by other processes. At the same time, the weights of each causal edge are combined for weighted propagation calculation. After calculating and ranking the root cause contributions of all process nodes, the process node with the highest root cause contribution value is determined as the root cause process of the anomaly, and other processes subsequently affected by it through causal edges are determined as affected processes on the anomaly propagation chain. Further reading of detailed metadata information of the root cause process in proc_meta_map, such as process name, startup time, parent process information, etc., and tracing back the key system call sequence and kernel operation records executed by the root cause process within the abnormal time window, can pinpoint the specific operational behavior that caused the abnormal event.

[0067] In alternative implementations, such as Figure 6 As shown, step S300 determines a troubleshooting strategy based on the root cause localization, and processes the abnormal event according to the troubleshooting strategy, including: S310: Based on the root cause localization result, obtain the identification information of the root cause process and the abnormal operation type.

[0068] S320: Input the identification information of the root cause process and the abnormal operation type into the strategy generation module. The strategy generation module queries the preset operation and maintenance knowledge base, matches the handling rules corresponding to the abnormal operation type, and generates a troubleshooting strategy. The troubleshooting strategy includes operation instructions for the root cause process, recovery instructions for the affected process, and adjustment instructions for system resources.

[0069] S330: According to the operation instructions in the troubleshooting strategy, perform termination or resource restriction operations on the root cause process, perform resource recovery or restart operations on the affected process, and perform quota adjustment or isolation operations on the system resources.

[0070] In this embodiment, a troubleshooting strategy is determined based on root cause localization, and abnormal events are handled according to the strategy. After root cause localization, the system first obtains the identification information and abnormal operation type of the root cause process. The root cause process identification information includes process PID, process name, and container identifier, etc., and the abnormal operation type includes categories such as batch process creation, memory resource consumption, CPU resource preemption, and file handle leakage. A preset operation and maintenance knowledge base is queried, which contains mapping records of various abnormal operation types and corresponding handling rules. Each rule includes an operation instruction template, execution parameters, priority, and risk level. After matching the corresponding rule, the placeholder variables in the template are replaced with the actual process identifier and resource limit value to generate a troubleshooting strategy. Optionally, the strategy may include three types of instructions: first, operation instructions for the root cause process, such as terminating the process or restricting its resource usage; second, recovery instructions for the affected process, such as restarting the process or restoring its resource quota; and third, adjustment instructions for system resources, such as modifying global resource parameters or performing resource isolation.

[0071] The execution module sequentially calls system interfaces or container runtime interfaces to execute instructions according to the strategy. For example, it terminates the root cause process via the process management interface, restricts process resource usage via the cgroup interface, restarts affected processes via the service management interface, and modifies system resource configuration via the kernel parameter adjustment interface. The result of each execution step is recorded and reported. After the strategy is executed, the data acquisition module continuously collects time-series indicators within the processed time window and re-inputs them into multiple sets of anomaly detection algorithms for secondary detection. If the overall anomaly score falls back to the normal range, the anomaly event is deemed successfully handled, the event status is updated to "handled," and the event is archived, forming a complete automated operation and maintenance closed loop.

[0072] This application also provides a single-node fault intelligent operation and maintenance system based on multi-algorithm fusion, such as... Figure 7 As shown, the system includes a data acquisition module 11, a kernel-mode tracing module 12, and a root cause localization module 13.

[0073] The data acquisition module 11 is used to collect time-series indicators of a single-node Linux operating system, perform anomaly detection on the time-series indicators using multiple anomaly detection algorithms, and perform fusion processing on the output results of the multiple anomaly detection algorithms to generate anomaly events. Kernel-mode tracing module 12 is used to collect kernel-mode process data within the time window corresponding to abnormal events through multiple eBPF subroutines based on graph construction logic, and to construct a kernel-mode causal dependency graph. The root cause localization module 13 is used to complete the root cause localization of abnormal events based on the causal dependency graph, determine the troubleshooting strategy based on the root cause localization, and process the abnormal events according to the troubleshooting strategy.

[0074] Those skilled in the art will understand that the technical solutions described in the embodiments of this application can also be applied to various single-node environments running the Linux operating system, such as virtual machines, edge computing nodes, and embedded devices, and are not limited to the physical servers and specific business scenarios described in the above embodiments. The eBPF subroutine splitting method, tail call chaining method, sharded storage architecture, and incremental update mechanism described in the embodiments of this application can all be adjusted and adapted according to the actual application scenario and node scale. For example, for edge computing nodes with a small number of processes, the number and capacity of shard mappings can be appropriately reduced. For complex scenarios that require monitoring more causal dimensions, additional causal edge computing subroutines can be added and stored in the reserved BPF program array mapping index position and inserted into the tail call chain. Adjustments to these implementation methods are still within the protection scope of this application.

[0075] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A single-node fault operation and maintenance method based on multi-algorithm fusion, characterized in that, include: Time-series metrics of a single-node Linux operating system are collected, and anomalies are detected by multiple sets of anomaly detection algorithms. The output results of the multiple sets of anomaly detection algorithms are fused to generate anomaly events. Kernel-mode process data within the time window corresponding to abnormal events are collected by multiple eBPF subroutines based on graph construction logic, and a kernel-mode causal dependency graph is constructed. The root cause of the abnormal event is located based on the causal dependency graph, a troubleshooting strategy is determined based on the root cause location, and the abnormal event is processed according to the troubleshooting strategy.

2. The fault operation and maintenance method according to claim 1, characterized in that, Constructing a kernel-mode causal dependency graph includes: The graph construction logic is broken down into multiple independent eBPF subroutines. Each eBPF subroutine performs a single atomic operation and passes the compliance check by the kernel verifier. Create a BPF program array mapping and store multiple eBPF subroutines into the corresponding index positions of the BPF program array mapping; The graph is constructed by chaining multiple eBPF subroutines in a preset order through eBPF tail calls.

3. The fault operation and maintenance method according to claim 2, characterized in that, The graph construction logic is broken down into multiple independent eBPF subroutines, including: The process metadata collection subroutine, lifecycle causal edge calculation subroutine, resource dependency causal edge calculation subroutine, IPC communication causal edge calculation subroutine, and graph adjacency matrix update subroutine are separated.

4. The fault operation and maintenance method according to claim 3, characterized in that, The graph construction is accomplished by concatenating multiple eBPF subroutines in a preset order through eBPF tail calls, including: After the process metadata collection subroutine completes the collection and writing of process metadata, it jumps to the lifecycle causal edge calculation subroutine through the first eBPF tail call. After the lifecycle causal edge calculation subroutine completes the calculation and writing of the lifecycle causal edge, it jumps to the resource dependency causal edge calculation subroutine through the second eBPF tail call. After the resource-dependent causal edge calculation subroutine completes the calculation and writing of the resource-dependent causal edge, it jumps to the IPC communication causal edge calculation subroutine through the third eBPF tail call. After the IPC communication causal edge calculation subroutine completes the calculation and writing of the IPC communication causal edge, it jumps to the graph adjacency matrix update subroutine through the fourth eBPF tail call. The graph adjacency matrix update subroutine performs a merge update of the global causal graph data to complete the graph construction.

5. The fault operation and maintenance method according to claim 4, characterized in that, The process of constructing the kernel-mode causal dependency graph also includes: Each eBPF subroutine is configured with an independent eBPF hash map as a sharded storage unit, and each sharded storage unit stores only the causal edge data generated by the corresponding eBPF subroutine. The complete causal graph data, merged by the graph adjacency matrix update subroutine, is stored through a global adjacency matrix mapping.

6. The fault operation and maintenance method according to claim 5, characterized in that, The process of constructing the kernel-mode causal dependency graph also includes updating the causal dependency graph through an event-driven incremental update mechanism: When the kernel captures a preset trigger event, it triggers the execution of the eBPF subroutine corresponding to the preset trigger event; Only the causal edge data in the corresponding shard storage unit of the eBPF subroutine is updated, and the subroutine updated through the graph adjacency matrix only merges the changed causal edge data into the global adjacency matrix mapping. The preset triggering events include process lifecycle events, resource operation events and IPC communication events.

7. The fault operation and maintenance method according to claim 2, characterized in that, The process of constructing the kernel-mode causal dependency graph also includes: The consistency of shared data is ensured by executing sequential locks and atomic operations during the serial execution of multiple eBPF subroutines in the tail call.

8. The fault operation and maintenance method according to claim 2, characterized in that, Also includes: Cross-kernel version adaptation is achieved through cross-platform compilation based on kernel BTF type information; The abnormal events are input into the large model enhancement layer, and combined with the root cause localization results, natural language explanations and handling suggestions are generated.

9. The fault operation and maintenance method according to claim 1, characterized in that, The step of determining a troubleshooting strategy based on the root cause localization and processing the abnormal event according to the troubleshooting strategy includes: Based on the root cause localization results, obtain the identification information and abnormal operation type of the root cause process; The identification information of the root cause process and the abnormal operation type are input into the strategy generation module. The strategy generation module queries the preset operation and maintenance knowledge base, matches the handling rules corresponding to the abnormal operation type, and generates a troubleshooting strategy. The troubleshooting strategy includes operation instructions for the root cause process, recovery instructions for the affected process, and adjustment instructions for system resources. According to the operation instructions in the troubleshooting strategy, terminate or limit the root cause process, restore or restart the affected process, and adjust or isolate the system resources.

10. A single-node fault intelligent operation and maintenance system based on multi-algorithm fusion, characterized in that, include: The data acquisition module is used to collect time-series indicators of a single-node Linux operating system, perform anomaly detection on the time-series indicators using multiple anomaly detection algorithms, and perform fusion processing on the output results of the multiple anomaly detection algorithms to generate anomaly events. The kernel-mode tracing module is used to collect kernel-mode process data within the time window corresponding to abnormal events through multiple eBPF subroutines based on graph construction logic, and to construct a kernel-mode causal dependency graph. The root cause localization module is used to locate the root cause of an abnormal event based on the causal dependency graph, determine a troubleshooting strategy based on the root cause localization, and process the abnormal event according to the troubleshooting strategy.

Images