A k8s business system four-dimensional index cooperative fault delimiting analysis method

Abstract

The application provides a k8s business system four-dimensional index cooperative fault delimitation analysis method, relates to the social network analysis technical field, and includes the following steps: constructing an application, network, resource and page fault kernel four-dimensional cooperative delimitation system based on the characteristics of the k8s containerization architecture, realizing the homologous collection and dimension alignment of the four-dimensional indexes through an eBPF probe, establishing a multidimensional correlation and time synchronization mechanism, combining an intelligent causal analysis rule engine to complete automatic fault delimitation, and outputting the delimitation result after low-loss data transmission and aggregation. The application realizes the automatic and accurate delimitation of fault causes, optimizes the system overhead through low-loss data transmission and multidimensional aggregation, finally outputs a standardized delimitation report and realizes visual display and alarm, comprehensively improves the efficiency, accuracy and refinement degree of k8s business system fault delimitation, reduces the operation and maintenance cost and system resource loss, and adapts to the high-frequency fault disposal demand of large-scale k8s clusters.

Description

Technical Field

[0001] This invention belongs to the field of container operation and maintenance technology, and relates to a four-dimensional indicator collaborative fault delimitation analysis method for Kubernetes business systems, and more particularly to a fault delimitation analysis method for Kubernetes business systems that coordinates kernel page fault anomaly indicators with observable indicators. Background Technology

[0002] In business systems deployed in k8s (Kubernetes) containers, fault delimitation analysis relies on the collaborative judgment of multi-dimensional observable indicators. Existing technologies generally adopt a three-dimensional delimitation system of "application indicators, network indicators, and resource indicators" to locate the root cause of faults through the linkage of multiple indicators. Specifically, it includes: (1) Application indicator monitoring: By deploying APM or eBPF tools in business applications, core indicators of the application layer are collected, including interface response latency, error rate, request volume, etc., to locate application faults. (2) Network indicator monitoring: By deploying collection probes at the k8s node network namespace, container network card, or service mesh (such as eBPF, NPM packet capture probe, etc.), core indicators of the network layer are collected, including retransmission rate, packet loss rate, latency, delay, request success rate, etc., to locate network topology and link transmission faults. (3) Resource metrics monitoring: Collect node / container level resource metrics, including CPU utilization, physical memory utilization, disk I / O throughput, container start and stop count, etc., through the k8s native metrics server (metrics-server) or third-party monitoring tools (such as Prometheus), to locate resource overload or unreasonable resource allocation faults.

[0003] The existing fault localization logic is as follows: The monitoring platform aggregates data from the three categories of applications, networks, and resources. When an anomaly occurs in the business, operations personnel manually filter and compare the time-series data and related dimensions (such as Pods and namespaces) of various indicators based on preset indicator correlation rules (e.g., judging resource overload as "increased interface response latency, 100% node CPU utilization") to attempt to identify the root cause of the fault. Its deployment architecture consists of three independent monitoring units deployed within the Kubernetes cluster: an APM tool, a network probe, and a resource monitoring component. These units report data to the cluster-level monitoring platform through different communication links. The platform provides independent display of multiple indicators and manual correlation analysis capabilities.

[0004] However, the three-dimensional delimitation system lacks core indicators. Page faults, a key kernel-level indicator reflecting the effectiveness of process memory access, are not included in the collaborative delimitation system. Existing solutions can only indirectly infer memory-related problems through macro-level resource indicators such as physical memory utilization, and cannot obtain micro-level anomalies in process memory access (such as process blocking caused by major page faults, business lag caused by memory-mapped file loading failures, etc.), making it difficult to accurately locate the coupling problem between memory and other dimensions of faults.

[0005] The few independent monitoring solutions for page faults also fail to achieve coordinated linkage with the three types of indicators mentioned above: one type is based on collecting the cumulative parameters of major and minor page faults in the task structure based on high-frequency kernel events such as thread switching and program execution, which has the problem of significant performance loss; another type is based on capturing page fault events based on general trace points, which has high data redundancy and is not associated with application, network, and resource indicators, and cannot be integrated into the existing delineation system, and can only be used as independent data reference.

[0006] Overall, the existing three-dimensional fault delimitation system of "application, network, and resources" has the following fatal flaws in the four-dimensional fault delimitation analysis of Kubernetes business systems due to the lack of kernel page fault anomaly indicators and coordination mechanisms:

[0007] (1) Lack of core indicators and low accuracy in delineating memory-related faults: Existing solutions rely solely on macro-level resource indicators such as physical memory utilization to infer memory problems, failing to capture micro-level kernel-level anomalies such as kernel page faults. When business anomalies are caused by memory access anomalies (such as a major page fault triggering process blocking, or a memory-mapped file loading failure causing data read delays), the lack of kernel page fault indicators makes it easy to misjudge as application code errors or network latency issues, leading to root cause misidentification.

[0008] (2) Poor synergy of indicators and ineffective handling of coupled faults: Memory anomalies are often coupled with application and network faults (such as a primary page fault causing process blocking and then triggering an interface timeout, or packet loss causing memory-mapped file loading failure and triggering a primary page fault). Existing solutions are unable to establish a complete causal chain of "memory micro-anomaly - application performance degradation - network link anomaly" due to the lack of kernel page fault indicators. When facing such coupled faults, the delimitation efficiency is extremely low, and it is even impossible to locate the root cause.

[0009] (3) Existing kernel page fault monitoring solutions suffer from high performance overhead and cannot be integrated into the existing system: A few independent kernel page fault monitoring solutions are based on high-frequency kernel events such as thread switching. In a multi-container, multi-threaded environment of k8s, they are triggered tens of thousands of times per second. When transmitting data through Unix Domain Sockets based on the file system, they are prone to communication congestion, data backlog, and occupy a lot of kernel resources. Moreover, the output data of these solutions is not associated with application, network, and resource indicators, and cannot be integrated into the existing delimitation system, forming data silos.

[0010] (4) Insufficient monitoring accuracy and lack of fine-grained correlation dimensions: Existing application and network indicator monitoring is mostly limited to the Pod or container level and cannot be extended to the thread level; while the kernel page fault exception independent monitoring solution can obtain the thread ID, it is not associated with fine-grained dimensions such as application interface and network connection, which makes it impossible to accurately locate "which business interface - which thread - which network connection" triggered the exception, and the delimitation range is too large.

[0011] (5) High dependence on manual intervention and low delineation efficiency: The existing solution requires maintenance personnel to manually compare the three types of indicators of application, network and resources with independent kernel page fault data (if any). This is not only time-consuming and laborious, but also prone to root cause location deviation due to human judgment errors. When facing high-frequency faults of large-scale k8s clusters, it cannot meet the maintenance needs of rapid delineation.

[0012] Therefore, there is an urgent need for a four-dimensional indicator collaborative fault delimitation analysis method for Kubernetes business systems to solve the defects of the three-dimensional delimitation system, such as the lack of kernel page fault anomaly indicators, poor indicator coordination, low delimitation accuracy, and low efficiency. This method can realize the linkage analysis of memory anomalies with other dimensional indicators, thereby improving the efficiency and accuracy of fault delimitation. Summary of the Invention

[0013] To address the problems in the background technology, this invention provides a four-dimensional indicator collaborative fault delimitation analysis method for Kubernetes business systems. It incorporates kernel page fault anomaly indicators into the Kubernetes fault delimitation system, constructs a four-dimensional collaborative architecture of application, network, resources, and kernel page fault anomaly kernel, realizes accurate linkage between memory micro-anomalies and other dimension faults, and ultimately achieves automatic fault delimitation.

[0014] To achieve the above objectives, this invention provides a four-dimensional index collaborative fault delimitation analysis method for Kubernetes business systems, comprising:

[0015] Deploy eBPF probes on Kubernetes cluster nodes. Based on the eBPF probes, kernel page fault error metric collection modules, application metric collection modules, network metric collection modules, and resource metric collection modules are used to complete the fine collection of four-dimensional metrics. The data collected by each module is associated with Kubernetes resource dimension identifiers and thread-level identifiers.

[0016] Using thread TID, container ID, and timestamp window as unified association keys, and combined with a nanosecond-level unified timestamp synchronization mechanism, thread-level fine-grained association and full-dimensional alignment of four-dimensional metrics with Kubernetes are achieved.

[0017] Based on kernel page fault error code parsing, page fault types are analyzed and user-mode primary page faults affecting business are filtered out. Four-dimensional indicator abnormal thresholds are preset, and an intelligent causal association rule engine is built. Causal association rules are matched according to the temporal triggering relationship and abnormal state of the four-dimensional indicators to automatically delineate the root cause of the fault.

[0018] Based on the fault delimitation results, a standardized fault delimitation report is generated. The four-dimensional indicator correlation data and the standardized delimitation report are then reported to the Kubernetes cluster-level monitoring platform. The monitoring platform displays the four-dimensional indicators and pushes anomaly alarms.

[0019] As a further improvement of the present invention, the eBPF probe is deployed non-intrusively through a kernel dynamic mounting mechanism, and the kernel page fault index collection module is accurately mounted to the dedicated tracking point of kernel page fault, including user-mode page fault and kernel-mode page fault. It adopts an event-driven mode to collect corresponding data only when a kernel page fault event is triggered.

[0020] As a further improvement of the present invention, the core data collected by the kernel page fault exception indicator collection module includes process ID, thread ID, page fault virtual address, address of the instruction that triggered the page fault, page fault error code, event timestamp, and container ID. The application indicator collection module collects the interface call information of the application process. The network indicator collection module collects indicators including TCP retransmission rate, cross-pod latency, and VXLAN latency. The resource indicator collection module collects indicators including process-level CPU utilization and memory usage.

[0021] As a further improvement of the present invention, the nanosecond-level unified timestamp synchronization mechanism includes: obtaining nanosecond-level timestamps by calling the kernel nanosecond-level time acquisition interface through the eBPF probe, uniformly marking all collected data of the four-dimensional indicators with time stamps, eliminating the time difference between different acquisition modules, and providing timing basis for thread-level fine-grained correlation.

[0022] As a further improvement of the present invention, the timestamp window can be dynamically configured through a Kubernetes Custom Resource Definition (CRD), with a default duration of 10ms;

[0023] The thread-level fine-grained association of the four-dimensional indicators includes: accurately matching and associating kernel page fault indicators, application indicators, network indicators, and resource indicators under the same thread TID, the same container ID, and within the same timestamp window, so that the four-dimensional indicators correspond one-to-one under the single thread dimension.

[0024] The Kubernetes full-dimensional alignment includes: parsing the cgroup control group file corresponding to the process, calling the Kubernetes API to supplement all metrics with Kubernetes resource dimension identifiers, including container ID, Pod name, namespace, and service name, to achieve full-link association between cluster, namespace, Pod, container, process, and thread.

[0025] As a further improvement of the present invention, the step of parsing the page fault type based on the kernel page fault exception error code includes:

[0026] The lower three flag bits of the error_code for a kernel page fault are parsed. The 0th bit determines the cause of the page fault: 0 for page not found, 1 for illegal permissions. The 1st bit determines the access type: 0 for read or execute, 1 for write. The 2nd bit determines the triggering context: 0 for kernel mode, 1 for user mode. Only major page faults in user mode where the page does not exist are selected as the core anomaly indicator for fault analysis.

[0027] As a further improvement of the present invention, the four-dimensional indicator anomaly threshold is dynamically configured through a Kubernetes custom resource definition, and the preset basic threshold includes:

[0028] User-mode primary page faults ≥5 per second are considered kernel page fault anomalies; application interface response latency ≥500ms or error rate ≥1% are considered application anomalies; TCP retransmission rate ≥3% or VXLAN encapsulation latency ≥100ms are considered network anomalies; and container CPU utilization ≥80% or physical memory utilization ≥90% are considered resource anomalies.

[0029] As a further improvement of the present invention, the causal association rules in the intelligent causal association rule engine include at least:

[0030] If a user-mode primary page fault is triggered first and occurs ≤50ms earlier than other faults, and resource metrics are normal, it is determined that a kernel page fault is causing a decrease in application performance.

[0031] If a network metric anomaly is triggered first and occurs ≤100ms earlier than other anomalies, or if the number of user-space main page faults increases within the same thread or container dimension, it is determined that a network fault has caused an abnormal kernel page fault metric.

[0032] An abnormal resource metric was triggered first, accompanied by an increase in the number of user-space main page faults. This was determined to be due to resource overload causing memory swapping, which in turn triggered a kernel page fault.

[0033] If only the application metrics are abnormal while other metrics are normal, it is determined to be a pure application code-level fault.

[0034] If only network metrics are abnormal while other metrics are normal, the problem is determined to be a pure network link failure.

[0035] As a further improvement of the present invention, the four-dimensional index data collected by the eBPF probe is transmitted to the user-space collector through a Unix domain socket communication mechanism of batch encapsulation and asynchronous transmission. Before transmission, the collected data is compressed using a data compression algorithm to achieve low-loss data transmission from kernel space to user space.

[0036] The user-mode collector performs multi-dimensional aggregation on the associated four-dimensional indicator data according to the Kubernetes resource dimension and the business dimension, and calculates the statistical value of abnormal indicators per unit time.

[0037] As a further improvement of the present invention, the standardized fault delimitation report includes fault root cause type, fault impact range, abnormal indicator details, and indicator time series correlation graph. The fault impact range includes associated Kubernetes resource identifiers and business interface information. The monitoring platform provides a four-dimensional indicator integrated visualization interface, supports filtering and querying by Kubernetes resource dimension and business interface dimension, and pushes abnormal alarms to the operation and maintenance system.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] This invention constructs a four-dimensional collaborative fault delimitation system encompassing application, network, resource, and kernel page fault anomalies. It utilizes eBPF probes to achieve refined collection of four-dimensional indicators and correlates them with Kubernetes resource dimension identifiers and thread-level identifiers. Combined with a unified association key and nanosecond-level time synchronization mechanism, it achieves fine-grained thread-level association and full-dimensional alignment with Kubernetes. Furthermore, an intelligent causal association rule engine automatically delimits the root causes of faults and outputs standardized reports and visualizations. This invention supplements the core indicators of kernel page fault anomalies missing in existing technologies, solving the problems of poor indicator coordination, low fault delimitation accuracy, and high dependence on manual intervention in traditional three-dimensional delimitation systems. It achieves accurate and automatic delimitation of the root causes of faults in Kubernetes business systems, significantly improving the efficiency and accuracy of fault delimitation.

[0040] This invention achieves thread-level fine-grained association of four-dimensional indicators, accurately matching four-dimensional indicators within the same thread TID, container ID, and timestamp window. It breaks through the limitation of existing technologies that can only achieve coarse-grained association at the Pod / container level, and can accurately locate the source of the fault in "k8s resource-process-thread-business interface". It solves the problems of insufficient monitoring accuracy and excessively large delimitation range in traditional solutions, and greatly improves the fineness of fault delimitation.

[0041] This invention deploys eBPF probes non-intrusively on Kubernetes cluster nodes through a kernel dynamic mounting mechanism, and precisely mounts the kernel page fault anomaly indicator collection module to dedicated tracking points for kernel user-mode page faults and kernel-mode page faults. It adopts an event-driven mode to collect data only when a page fault event is triggered. Compared with existing kernel page fault anomaly monitoring solutions based on high-frequency kernel events, this significantly reduces the performance loss of indicator collection on the system, and does not require restarting nodes, modifying the kernel, or interrupting services, thus adapting to the operation and maintenance requirements of uninterrupted operation of Kubernetes business systems.

[0042] This invention extracts container identifiers by parsing the cgroup control group kernel virtual file and calls the k8s API to complete the k8s resource identifiers. It achieves a full-link association between cluster, namespace, Pod, container, process, and thread, making kernel / network underlying metrics without business attributes and application metrics with business attributes form a unified dimension. This allows fault delimitation results to be directly applied to specific k8s business resources, solving the problem of fragmented metric dimensions in existing technologies and making the delimitation results more valuable for practical operation and maintenance.

[0043] This invention transmits four-dimensional indicator data collected by eBPF probes to the user-space collector via a Unix domain socket communication mechanism of batch encapsulation and asynchronous transmission. The data is compressed using the LZ4 algorithm, and the associated data is aggregated in multiple dimensions according to Kubernetes resources and business interfaces. Compared with existing traditional socket transmission schemes, this avoids communication congestion and data backlog problems caused by high-frequency small packet transmission, further reducing the system overhead of data transmission. Moreover, the aggregated structured data is more convenient for subsequent fault analysis and visualization, and is suitable for the monitoring needs of large-scale Kubernetes clusters.

[0044] The four-dimensional indicator anomaly threshold of this invention is dynamically configured through Kubernetes custom resource definition. The intelligent causal association rule engine can automatically match rules for delimitation based on the indicator's temporal triggering relationship and abnormal state. The fault delimitation report is clearly specified to the associated Kubernetes resources, business interfaces, and abnormal triggering threads. The monitoring platform supports multi-dimensional filtering and querying and automatically pushes alarms. Compared with the existing technology that relies on maintenance personnel to manually compare and analyze indicators, it greatly reduces manual intervention, improves the efficiency of handling high-frequency faults in large-scale Kubernetes clusters, and enables maintenance personnel to quickly locate faults and take targeted solutions. Attached Figure Description

[0045] Figure 1 This is a flowchart of a four-dimensional index collaborative fault delimitation analysis method for a Kubernetes business system disclosed in one embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] The present invention will now be described in further detail with reference to the accompanying drawings:

[0048] like Figure 1 As shown, the four-dimensional index collaborative fault delimitation analysis method for Kubernetes business systems provided by this invention specifically includes:

[0049] S1. Deploy eBPF probes on Kubernetes cluster nodes. Based on the eBPF probes, kernel page fault anomaly indicator collection modules, application indicator collection modules, network indicator collection modules, and resource indicator collection modules are used to complete the fine-grained collection of four-dimensional indicators. The data collected by each module is associated with Kubernetes resource dimension identifiers and thread-level identifiers. This step achieves non-intrusive collection of four-dimensional indicators from the same source through eBPF probes. Combined with optimized transmission and aggregation methods, it not only completes the core indicators of kernel page fault anomalies but also significantly reduces the system performance loss in the collection and transmission process, laying an accurate data foundation for subsequent indicator correlation analysis.

[0050] Among them, the eBPF probe is deployed non-intrusively through the kernel dynamic mounting mechanism. The kernel page fault index collection module is accurately mounted to the dedicated tracking point of kernel page fault, including user-mode page fault and kernel-mode page fault. It adopts an event-driven mode and only collects corresponding data when the kernel page fault event is triggered.

[0051] Furthermore, the core data collected by the kernel page fault exception collection module includes process ID, thread ID, page fault virtual address, address of the instruction that triggered the page fault, page fault error code, event timestamp, and container ID. The application metric collection module collects the interface call information of the application process. The network metric collection module collects metrics including TCP retransmission rate, cross-pod latency, and VXLAN latency. The resource metric collection module collects metrics including process-level CPU utilization and memory usage.

[0052] The four-dimensional indicator data collected by the eBPF probe is transmitted to the user-space collector through a Unix domain socket communication mechanism of batch encapsulation and asynchronous transmission. The collected data is compressed using a data compression algorithm before transmission to achieve low-loss data transmission from kernel space to user space.

[0053] The user-mode data collector performs multi-dimensional aggregation on the associated four-dimensional indicator data according to the Kubernetes resource dimension and business dimension, and calculates the statistical value of abnormal indicators per unit time.

[0054] Specifically, the eBPF probe is deployed as a DaemonSet resource in the Kubernetes cluster, enabling unified deployment and maintenance of probes across all nodes in the cluster. The kernel page fault metric collection module defines the `page_fault_data` data structure by writing an eBPF program, which includes core collection fields such as process ID, thread ID, and page fault virtual address. It also creates a `page_fault_map` mapping table of type `BPF_MAP_TYPE_PERF_EVENT_ARRAY`, and sends the collected data to user space through this mapping table. The application metric collection module attaches the eBPF probe to the function call of the application process. The entry point collects information such as interface name and request parameters and strongly associates them with thread TID. The network metric collection module uses eBPF probes to attach to network tracing points such as tcp_sendmsg, tcp_recvmsg, and tcp_retransmit_skb, as well as XDP hook points. It writes a net_metric_data data structure to collect network metrics and associates them with thread ID (TID) and Kubernetes resource information. The resource metric collection module uses eBPF probes to read kernel task structures and cgroup control group resource limit information, collects CPU and memory metrics from the process dimension and associates them down to the thread dimension. The data compression algorithm adopts the LZ4 algorithm. The trigger condition for batch encapsulation and transmission is the accumulation of 10 associated data. The user-space collector performs data aggregation according to six dimensions: namespace-Pod-container-process-thread-business interface, and counts the number of abnormal metrics, average values, and other statistical values ​​per unit time to generate structured data.

[0055] This invention takes TCP retransmission data collection as an example, and the core code snippet includes:

[0056] / / Define the network metric data structure (associating thread TID with Kubernetes resources)

[0057] struct net_metric_data {

[0058] __u32 pid; / / Process ID

[0059] __u32 tid; / / Thread ID

[0060] __u64 tcp_retrans; / / Number of TCP retransmissions

[0061] __u64 conn_delay; / / Connection establishment delay (nanoseconds)

[0062] __u64 vxlan_delay; / / VXLAN latency (nanoseconds)

[0063] __u64 timestamp; / / Event timestamp (nanoseconds)

[0064] char container_id

[64] ; / / Container ID

[0065] char pod_name

[64] ; / / Pod name

[0066] char service_name

[64] ; / / Associate Service name

[0067] };

[0068] struct bpf_map_def SEC("maps / net_metric_map") net_metric_map = {

[0069] .type = BPF_MAP_TYPE_PERF_EVENT_ARRAY,

[0070] .key_size = sizeof(__u32),

[0071] .value_size = sizeof(__u32),

[0072] .max_entries = 1024,

[0073] };

[0074] / / TCP retransmission event collection (tracepoint: tcp_retransmit_skb)

[0075] SEC("tracepoint / tcp / tcp_retransmit_skb")

[0076] int BPF_PROG(tcp_retrans_collect, struct sock *sk) {

[0077] struct net_metric_data data = {0};

[0078] struct task_struct *task = (struct task_struct *)bpf_get_current_task();

[0079] data.pid = bpf_get_current_pid_tgid() >> 32;

[0080] data.tid = bpf_get_current_pid_tgid() & 0xFFFFFFFF;

[0081] data.tcp_retrans = 1; / / Single retransmission count

[0082] data.timestamp = bpf_ktime_get_ns();

[0083] / / Associate container / Pod / Service information

[0084] bpf_get_current_cgroup_path(&data.container_id, sizeof(data.container_id), 0);

[0085] bpf_probe_read_user_str(&data.pod_name, sizeof(data.pod_name), (void*)task->comm);

[0086] bpf_get_service_by_sock(sk,&data.service_name, sizeof(data.service_name)); / / Custom helper function to associate via sock

[0087] / / Send data to user space

[0088] bpf_perf_event_output(sk, &net_metric_map, BPF_F_CURRENT_CPU, &data,sizeof(data));

[0089] return 0;

[0090] }

[0091] S2 uses thread TID, container ID, and timestamp window as unified association keys, combined with a nanosecond-level unified timestamp synchronization mechanism, to perform thread-level fine-grained association of four-dimensional metrics and full-dimensional alignment with Kubernetes; it breaks through the limitations of traditional Pod / container-level coarse-grained association, eliminates the time deviation of multi-module collection, and enables the underlying metrics to be accurately bound to Kubernetes business resources.

[0092] The nanosecond-level unified timestamp synchronization mechanism includes: obtaining nanosecond-level timestamps by calling the kernel's ktime_get_ns() nanosecond-level time acquisition interface through the eBPF probe, uniformly marking all collected data of the four-dimensional indicators with time stamps, eliminating time differences between different acquisition modules, and providing timing basis for thread-level fine-grained correlation.

[0093] Furthermore, the timestamp window can be dynamically configured through Kubernetes custom resource definitions, with a default duration of 10ms;

[0094] Perform thread-level fine-grained association on four-dimensional metrics, including: accurately matching and associating kernel page fault metrics, application metrics, network metrics, and resource metrics under the same thread TID, the same container ID, and within the same timestamp window, so that the four-dimensional metrics correspond one-to-one under a single thread dimension;

[0095] Kubernetes full-dimensional alignment includes: parsing the cgroup control group file corresponding to the process, calling the Kubernetes API to supplement all metrics with Kubernetes resource dimension identifiers, including container ID, Pod name, namespace, and service name, to achieve full-link association between cluster, namespace, Pod, container, process, and thread.

[0096] The user-space collector aggregates the associated four-dimensional indicator data in multiple dimensions according to the Kubernetes resource dimension and the business dimension, and calculates the statistical values ​​of abnormal indicators per unit time (such as the number of primary page faults per minute, average interface response latency, and TCP retransmission rate).

[0097] Specifically, nanosecond-level timestamps are obtained in real-time by the eBPF probe calling the `ktime_get_ns()` nanosecond-level time acquisition interface when collecting each metric data, and are written as core fields into the data structure of each metric. All collection modules use this method to achieve timestamp uniformity. Thread-level fine-grained association is achieved by writing an association algorithm in the user-space collector, using thread TID and container ID as the joint primary key and timestamp window as the time range, to perform precise matching on the four-dimensional metric dataset, filtering out metric data with the same primary key and the same time window to form an associated dataset. The parsed cgroup control group file is the ` / proc / [PID] / cgroup` virtual file generated by the Linux kernel for each process. By parsing this file, the unique identifier of the container is extracted, and then the k8s API is called based on this container identifier. The Server's Pod query interface retrieves the Kubernetes resource identifiers, such as the Pod name, namespace, and service name of the container, and adds these identifiers to all associated metric data. The Kubernetes CRD is configured as a custom FaultAnalysisConfig resource. By modifying the configuration parameters of this resource, parameters such as timestamp window duration and data aggregation period can be dynamically adjusted without restarting the probe or collector.

[0098] S3. Based on the kernel page fault error code parsing, page fault types are analyzed and user-mode primary page faults affecting business are selected. Preset four-dimensional indicator anomaly thresholds are established, and an intelligent causal association rule engine is constructed. Causal association rules are matched based on the temporal triggering relationship and abnormal state of the four-dimensional indicators to automatically delineate the root cause of the fault. This step, through accurate parsing of kernel page fault types and the construction of an intelligent causal association rule engine, achieves automated delineation of the root cause of the fault, establishes a causal association chain between indicators, and solves the problems of excessive manual intervention, low delineation efficiency, and inaccurate location of coupled faults in traditional solutions.

[0099] The page fault type parsing based on kernel page fault error codes includes: parsing the lower three flag bits of the kernel page fault error code. The 0th bit determines the cause of the page fault (0 for page not existing, 1 for illegal permissions); the 1st bit determines the access type (0 for read or execute, 1 for write); and the 2nd bit determines the triggering context (0 for kernel mode, 1 for user mode). Only user-mode primary page faults where the page does not exist are selected as the core anomaly indicators for fault analysis.

[0100] Furthermore, the four-dimensional indicator abnormality thresholds are dynamically configured through Kubernetes custom resource definitions. The preset basic thresholds include: ≥5 user-space primary page faults per second are considered kernel page fault abnormalities; ≥500ms application interface response latency or ≥1% error rate are considered application abnormalities; ≥3% TCP retransmission rate or ≥100ms VXLAN encapsulation latency are considered network abnormalities; and ≥80% container CPU utilization or ≥90% physical memory utilization are considered resource abnormalities.

[0101] The causal association rules in the intelligent causal association rule engine include at least the following:

[0102] (1) If a user-mode primary page fault is triggered first and ≤50ms earlier than other faults, and the resource indicators are normal, it is determined that the application performance is degraded due to a kernel page fault.

[0103] (2) When network indicator anomalies are triggered first and earlier than other anomalies by ≤100ms, and the number of user-mode main page faults increases in the same thread or container dimension, it is determined that the kernel page fault anomaly is caused by network failure.

[0104] (3) The abnormal resource indicators are triggered first, and the number of user-mode main page faults increases. It is determined that the memory replacement is caused by resource overload, which leads to kernel page faults.

[0105] (4) If only the application indicator is abnormal while other indicators are normal, it is determined to be a pure application code-level fault;

[0106] (5) If only network indicators are abnormal while other indicators are normal, it is determined to be a pure network link failure.

[0107] Specifically, the filtering of user-space primary page faults involves writing a parsing program in the user-space collector to perform bitwise operations on the error code field in the kernel page fault anomaly indicators, filtering out non-business-related page fault events such as kernel-space page faults and page faults caused by illegal permissions, and retaining only user-space primary page fault data for subsequent fault analysis. The anomaly determination of four-dimensional indicators involves the collector calculating the real-time value of the indicator and comparing it with a preset threshold. When the indicator value exceeds the threshold for three consecutive collection cycles, it is determined to be an indicator anomaly, and the anomaly trigger time is marked. The intelligent causal association rule engine is developed based on a rule engine framework, with built-in time-series analysis and rule matching modules. The time-series analysis module compares the trigger times of various anomaly indicators to determine the order of indicator anomalies. The rule matching module accurately matches the time-series relationship, anomaly dimension, indicator association status with preset causal association rules, and automatically outputs the root cause type of the fault upon successful matching. Dynamically configured anomaly thresholds defined by Kubernetes custom resources are stored in the fault analysis configuration resource, supporting individual configuration of personalized thresholds for different namespaces and different Pods, enabling refined anomaly determination.

[0108] S4. Generate a standardized fault delimitation report based on the fault delimitation results. Upload the four-dimensional indicator correlation data and the standardized delimitation report to the Kubernetes cluster-level monitoring platform. The monitoring platform displays the four-dimensional indicators and pushes anomaly alarms. This step transforms the fault delimitation results into a standardized report and enables visualization and automatic alarms, allowing the fault delimitation results to directly match the operating habits of Kubernetes operations and maintenance personnel, realizing a closed loop from technical indicator anomalies to actual operation and maintenance handling, and significantly improving fault handling efficiency.

[0109] The standardized fault delimitation report includes the fault root cause type, fault impact range, abnormal indicator details, and indicator time series correlation graph. The fault impact range includes the associated Kubernetes resource identifier and business interface information. The monitoring platform provides an integrated visualization interface for four-dimensional indicators, supports filtering and querying by Kubernetes resource dimension and business interface dimension, and pushes abnormal alarms to the operation and maintenance system.

[0110] Specifically, the standardized fault delimitation report is automatically generated by the user-space collector according to a fixed format. The fault root cause type is the delimitation result output by the rule engine. The fault impact range is accurate to the namespace, Pod, container, thread, and business interface where the anomaly was triggered. The anomaly indicator details include the anomaly indicator name, anomaly value, threshold, and trigger time. The indicator time-series correlation graph is based on a time axis, showing the anomaly triggering order and numerical change trend of each related indicator. The collector reports the four-dimensional indicator correlation data and the standardized delimitation report to the Kubernetes cluster-level monitoring platform through the Prometheus Remote Write protocol. The monitoring platform is built on Grafana to create an integrated visualization interface for four-dimensional indicators, providing functions such as multi-dimensional filtering and querying, indicator trend display, and fault report viewing. Anomaly alarm push is implemented through the alarm rules configured on the monitoring platform. When a fault delimitation result is detected, the alarm information and fault delimitation report are automatically pushed to operation and maintenance systems such as DingTalk, WeChat Work, and Prometheus Alarm Manager. The alarm information includes core content such as fault root cause, impact range, and handling suggestions, supporting operation and maintenance personnel to respond and handle quickly.

[0111] Example:

[0112] A company deployed a four-dimensional index collaborative fault delimitation and analysis system based on this invention in its production environment Kubernetes cluster. The specific implementation process is as follows:

[0113] Step 1, Deployment and Data Collection: Operations personnel deploy eBPF probes on all nodes of the cluster using Kubernetes DaemonSet. The probes are dynamically mounted to tracking points such as user-space page faults, kernel-space page faults, and TCP retransmission events via the kernel dynamic mounting mechanism. Real-time collection of kernel page fault metrics, application interface metrics, network metrics, and process resource metrics is achieved. All metrics are automatically associated with thread TID, container ID, and nanosecond-level timestamps, and efficiently transmitted to the user-space collector using batch encapsulation, asynchronous Unix domain sockets, and LZ4 compression.

[0114] Step 2, Metric Association and Dimension Alignment: The user-space collector uses thread TID, container ID, and 10ms timestamp window as unified association keys to perform thread-level fine-grained association of four-dimensional metrics within the same thread, container, and time window; at the same time, it parses the control group file to obtain the container ID, calls the k8s API to complete the Pod, namespace, and service name, and completes the full-link dimension alignment of cluster → namespace → Pod → container → process → thread.

[0115] Step 3, Anomaly Detection and Automatic Bounding: The system analyzes kernel page fault error codes in real time and filters out user-mode main page faults; it judges according to preset thresholds: when a thread in a Pod experiences 8 user-mode main page faults per second (exceeding the threshold of 5 times), and the anomaly is triggered 40ms earlier than the application interface timeout (>500ms), while the CPU and memory resource indicators are normal.

[0116] The intelligent causal association rule engine automatically matches rules and determines the root cause of the fault as: a thread triggers a kernel page fault, resulting in a decrease in application interface performance.

[0117] Step 4, Results Output and Alarms: The system automatically generates a standardized fault delimitation report, including the root cause of the fault, the abnormal thread ID, the Pod / namespace to which it belongs, the interface name, and the time series graph of the indicators; and reports the report and related data to the monitoring platform. The platform automatically pushes alarms to DingTalk and the alarm manager, so that operation and maintenance personnel do not need to manually investigate and can directly obtain accurate fault location and handling basis.

[0118] Advantages of this invention:

[0119] This invention is the first to incorporate kernel page fault indicators into the Kubernetes fault delimitation system, constructing a four-dimensional collaborative architecture of "application-network-resource-kernel page fault". It breaks through the limitations of the traditional three-dimensional system that relies solely on macro-level resource indicators to infer memory problems. It can accurately capture micro-level memory anomalies, establish a multi-indicator causal chain, supplement core indicators, effectively solve the problem of difficulty in delimiting coupled faults between memory and applications and networks, achieve accurate delimitation of coupled faults, and significantly improve the accuracy of delimiting memory-related faults.

[0120] This invention uses thread TID and container ID as the core association keys to achieve thread-level fine-grained association of four-dimensional indicators, breaking through the limitations of existing Pod / container-level coarse-grained positioning. Combined with Kubernetes full-dimensional link alignment, it accurately binds underlying indicators to business resources such as clusters, namespaces, and Pods, which can directly locate the source of the fault in "Kubernetes resources-business interfaces-threads", greatly reducing the scope of fault delimitation and investigation.

[0121] This invention constructs an intelligent causal association rule engine based on the temporal relationship of indicators. Combined with dynamically configurable anomaly thresholds, it realizes the automatic delineation of the root cause of the fault, replacing the traditional method of manually screening and comparing indicators, avoiding human judgment bias, significantly improving the delineation efficiency of high-frequency faults in large-scale Kubernetes clusters, and significantly reducing manual dependence and operation and maintenance costs.

[0122] This invention achieves non-intrusive probe deployment based on the eBPF kernel dynamic mounting mechanism. It only mounts dedicated trace points for kernel page fault exceptions to collect data in an event-driven mode, significantly reducing performance loss in the data collection process. Combined with batch encapsulation and asynchronous transmission of Unix domain socket communication and data compression, it avoids communication congestion and data backlog. The entire process does not require restarting nodes or interrupting services, and it is suitable for the uninterrupted operation requirements of Kubernetes multi-container and large-scale clusters.

[0123] This invention generates a standardized delimitation report that includes the root cause of the fault, the scope of impact, and the time series graph, accurately pinpointing the fault information to specific Kubernetes business resources. The supporting cluster-level monitoring platform provides integrated visualization of four-dimensional indicators and multi-dimensional filtering and querying. Anomaly alarms are automatically pushed to the operation and maintenance system, allowing technical indicator anomalies to be directly transformed into operation and maintenance instructions that can be quickly handled, realizing a closed-loop process from fault detection to delimitation and handling.

[0124] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for four-dimensional index-based collaborative fault delimitation analysis of a Kubernetes business system, characterized in that, include: Deploy eBPF probes on Kubernetes cluster nodes. Based on the eBPF probes, kernel page fault error metric collection modules, application metric collection modules, network metric collection modules, and resource metric collection modules are used to complete the fine collection of four-dimensional metrics. The data collected by each module is associated with Kubernetes resource dimension identifiers and thread-level identifiers. Using thread TID, container ID, and timestamp window as unified association keys, and combined with a nanosecond-level unified timestamp synchronization mechanism, thread-level fine-grained association and full-dimensional alignment of four-dimensional metrics with Kubernetes are achieved. Based on kernel page fault error code parsing, page fault types are analyzed and user-mode primary page faults affecting business are filtered out. Four-dimensional indicator abnormal thresholds are preset, and an intelligent causal association rule engine is built. Causal association rules are matched according to the temporal triggering relationship and abnormal state of the four-dimensional indicators to automatically delineate the root cause of the fault. Based on the fault delimitation results, a standardized fault delimitation report is generated. The four-dimensional indicator correlation data and the standardized fault delimitation report are then reported to the Kubernetes cluster-level monitoring platform. The monitoring platform displays the four-dimensional indicators and pushes anomaly alarms.

2. The method for four-dimensional index collaborative fault delimitation analysis of Kubernetes business systems according to claim 1, characterized in that: The eBPF probe is deployed non-intrusively through a kernel dynamic mounting mechanism. The kernel page fault index collection module is precisely mounted to the dedicated tracking point for kernel page faults, including user-mode page faults and kernel-mode page faults. It adopts an event-driven mode and collects corresponding data only when a kernel page fault event is triggered.

3. The method for four-dimensional index collaborative fault delimitation analysis of Kubernetes business systems according to claim 1, characterized in that: The core data collected by the kernel page fault exception indicator collection module includes process ID, thread ID, page fault virtual address, address of the instruction that triggered the page fault, page fault error code, event timestamp, and container ID. The application indicator collection module collects the interface call information of the application process. The network indicator collection module collects indicators including TCP retransmission rate, cross-pod latency, and VXLAN latency. The resource indicator collection module collects indicators including process-level CPU utilization and memory usage.

4. The method for four-dimensional index collaborative fault delimitation analysis of Kubernetes business systems according to claim 1, characterized in that, The nanosecond-level unified timestamp synchronization mechanism includes: obtaining nanosecond-level timestamps by calling the kernel nanosecond-level time acquisition interface through the eBPF probe, uniformly marking all collected data of the four-dimensional indicators with time stamps, eliminating the time difference between different acquisition modules, and providing timing basis for thread-level fine-grained correlation.

5. The method for four-dimensional index collaborative fault delimitation analysis of Kubernetes business systems according to claim 1, characterized in that: The timestamp window is dynamically configured through a Kubernetes custom resource definition, with a default duration of 10ms. The thread-level fine-grained association of the four-dimensional indicators includes: accurately matching and associating kernel page fault indicators, application indicators, network indicators, and resource indicators under the same thread TID, the same container ID, and within the same timestamp window, so that the four-dimensional indicators correspond one-to-one under the single thread dimension. The Kubernetes full-dimensional alignment includes: parsing the control group file corresponding to the process, calling the Kubernetes API to supplement all metrics with Kubernetes resource dimension identifiers, including container ID, Pod name, namespace, and service name, to achieve full-link association between cluster, namespace, Pod, container, process, and thread.

6. The method for four-dimensional index collaborative fault delimitation analysis of Kubernetes business systems according to claim 1, characterized in that, The method for parsing page fault types based on kernel page fault error codes includes: Based on the parsing of the lower three flag bits of the page fault error code, the 0th bit determines the cause of the page fault: 0 for page not existing and 1 for illegal permissions; the 1st bit determines the access type: 0 for read or execute and 1 for write; the 2nd bit determines the triggering context: 0 for kernel mode and 1 for user mode. Only major page faults in user mode where the page does not exist are selected as the core anomaly indicator for fault analysis.

7. The method for four-dimensional index collaborative fault delimitation analysis of Kubernetes business systems according to claim 1, characterized in that, The abnormal thresholds for the four-dimensional indicators are dynamically configured through Kubernetes custom resource definitions. The preset basic thresholds include: User-mode primary page faults ≥5 per second are considered kernel page fault anomalies; application interface response latency ≥500ms or error rate ≥1% are considered application anomalies; TCP retransmission rate ≥3% or VXLAN encapsulation latency ≥100ms are considered network anomalies; and container CPU utilization ≥80% or physical memory utilization ≥90% are considered resource anomalies.

8. The method for four-dimensional index collaborative fault delimitation analysis of Kubernetes business systems according to claim 1, characterized in that, The causal association rules in the intelligent causal association rule engine include at least the following: If a user-mode primary page fault is triggered first and occurs ≤50ms earlier than other faults, and resource metrics are normal, it is determined that a kernel page fault is causing a decrease in application performance. If a network metric anomaly is triggered first and occurs ≤100ms earlier than other anomalies, or if the number of user-space main page faults increases within the same thread or container dimension, it is determined that a network fault has caused an abnormal kernel page fault metric. An abnormal resource metric was triggered first, accompanied by an increase in the number of user-space main page faults. This was determined to be due to resource overload causing memory swapping, which in turn triggered a kernel page fault. If only the application metrics are abnormal while other metrics are normal, it is determined to be a pure application code-level fault. If only network metrics are abnormal while other metrics are normal, the problem is determined to be a pure network link failure.

9. The method for collaborative fault delimitation analysis of four-dimensional indicators in a Kubernetes business system according to claim 1, characterized in that: The four-dimensional index data collected by the eBPF probe is transmitted to the user-space collector through a Unix domain socket communication mechanism of batch encapsulation and asynchronous transmission. Before transmission, the collected data is compressed using a data compression algorithm to achieve low-loss data transmission from kernel space to user space. The user-mode collector performs multi-dimensional aggregation on the associated four-dimensional indicator data according to the Kubernetes resource dimension and the business dimension, and calculates the statistical value of abnormal indicators per unit time.

10. The method for four-dimensional index collaborative fault delimitation analysis of Kubernetes business systems according to claim 1, characterized in that: The standardized fault delimitation report includes the fault root cause type, fault impact range, abnormal indicator details, and indicator time-series correlation graph. The fault impact range includes the associated Kubernetes resource identifier and business interface information. The monitoring platform provides an integrated visualization interface for four-dimensional indicators, supports filtering and querying by Kubernetes resource dimension and business interface dimension, and pushes abnormal alarms to the operation and maintenance system.

Images