Container abnormality cause determination method and electronic device

Abstract

This application discloses a method and electronic device for determining the cause of container anomalies, relating to the field of data processing technology. The method includes: using a first probe to collect resource indicator values ​​of each container within a target node, triggering a second probe when an abnormal container exists; using the second probe to trace the resource access kernel call stack of the target node; determining multiple candidate associated containers based on the resource access kernel call stack and abnormal resource indicators; determining the target associated container and its corresponding target associated resource type based on the correlation between each candidate associated container and the abnormal container, and activating a third probe; using the third probe, based on the target associated resource type, determining the target associated resource access events with call stacks between the abnormal container and the target associated container within a preset time window, thereby determining the resource contention chain of the abnormal container and the cause of the abnormal container's anomaly. This achieves the technical effect of improving the accuracy of the determined anomaly cause.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and electronic device for determining the cause of container anomalies. Background Technology

[0002] As containerized deployment becomes the mainstream cloud-native approach, monitoring container runtime status is crucial for the normal operation of nodes. Sharing node resources among containers can easily lead to resource contention and noisy neighbor issues, causing performance fluctuations or even failures in containerized applications. Related technologies determine whether containers are experiencing anomalies by monitoring node-level or container-level resource metrics. However, monitoring node-level resource metrics cannot accurately identify abnormal containers, and therefore cannot determine the cause of the anomaly. While monitoring container-level resource metrics can identify abnormal containers, manually analyzing the anomaly based on container-level resource metrics after identification results in low accuracy of the determined cause. Summary of the Invention

[0003] This application provides a method for determining the cause of container anomalies, which at least solves the problem in related technologies that the cause of an anomaly in an abnormal container cannot be determined or the accuracy of the obtained cause of the anomaly is low.

[0004] This application provides a method for determining the cause of container anomalies, including: The first probe is used to collect resource index values ​​of each container within the target node; If any resource indicator value of any container is abnormal, identify the abnormal container and its abnormal resource indicators, and activate the second probe. Using the second probe, the kernel call stack of resource access of the target node is traced; based on the kernel call stack of resource access and abnormal resource indicators, multiple candidate associated containers are identified; the degree of association between each candidate associated container and the abnormal container is obtained; based on the degree of association, at least one target associated container and the target associated resource type corresponding to each target associated container are identified, and the third probe is activated. Using a third probe, based on the target associated resource type corresponding to each target associated container, the system determines the target associated resource access events with call stacks for the abnormal container and each target associated container within a preset time window. Based on the target associated resource access events with call stacks, the system determines the resource contention chain of the abnormal container, enabling users to determine the cause of the abnormality of the abnormal container based on the resource contention chain.

[0005] This application also provides a device for determining the cause of container anomalies, including: The acquisition module is used to collect resource index values ​​of each container within the target node using the first probe; The first determination module is used to determine the abnormal container and its abnormal resource indicators when any resource indicator value of any container is abnormal, and to enable the second probe. The second determination module is used to use the second probe to track the resource access kernel call stack of the target node; based on the resource access kernel call stack and abnormal resource indicators, determine multiple candidate associated containers; obtain the correlation degree between each candidate associated container and the abnormal container; based on the correlation degree, determine at least one target associated container and the target associated resource type corresponding to each target associated container, and enable the third probe. The third determination module is used to use a third probe to determine the access events of the abnormal container and each target associated container with the call stack within a preset time window based on the target associated resource type corresponding to each target associated container. Based on the access events of the target associated resource with the call stack, the resource contention chain of the abnormal container is determined, so that the user can determine the cause of the abnormality of the abnormal container based on the resource contention chain.

[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described methods for determining the cause of container anomalies when executing the computer program.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described methods for determining the cause of container exceptions.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described methods for determining the cause of container exceptions.

[0009] This application achieves deep tracing from container anomaly indicators to kernel call stack, and then to cross-container resource access events. By constructing a "resource contention link," the resource contention relationship and call path between the anomaly container and associated containers are clearly revealed. This transforms anomaly cause analysis from relying on manual experience to automated and visualized diagnosis based on objective call chain data, significantly improving the accuracy and efficiency of anomaly cause localization. Therefore, it solves the technical problems of being unable to determine anomaly causes and the low accuracy of obtained anomaly causes in related technologies, thereby improving the accuracy of anomaly cause localization. Thus, it addresses the technical problems of being unable to determine anomaly causes and the low accuracy of obtained anomaly causes in related technologies, achieving the technical effect of improving the accuracy of determined anomaly causes. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a method for determining the cause of container anomalies provided in an embodiment of this application; Figure 2 A flowchart illustrating another method for determining the cause of container anomalies provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of a container anomaly cause determination system provided in an embodiment of this application; Figure 4 This application provides a hierarchical probe state transition diagram for embodiments of the invention. Figure 5 This is a schematic diagram of a device for determining the cause of container anomalies provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0013] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0014] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] As containerized deployment becomes the mainstream cloud-native approach, monitoring container runtime status is crucial for the normal operation of nodes. In container runtime environments with shared kernels managed by container orchestration platforms such as Kubernetes, containers share node resources such as CPU, memory, I / O, and cache, which can easily lead to resource contention and noisy neighbor issues, causing performance fluctuations or even failures in containerized applications. Related technologies use node-level or container-level resource metrics, such as CPU utilization, control group (Cgroup) metrics, and network traffic, to determine if containers are experiencing anomalies. However, monitoring node-level resource metrics is costly and difficult to correlate with specific containers, making it difficult to accurately identify anomalous containers and precisely attribute underlying resource contention events to specific containers. Consequently, it is impossible to quickly and globally explain which neighboring container's behavior caused the slowdown, i.e., it is impossible to determine the cause of the anomalous container's anomaly. Monitoring container-level resource metrics is implemented using static probes from the Extended Berkeley Packet Filter (eBPF). Alarms are triggered by combining pre-set resource metric thresholds to identify anomalous containers. After identifying anomalous containers, manual analysis of the anomalies based on container-level resource metrics is required. However, this approach cannot directly link the anomalies to specific cross-container resource contention chains, resulting in low accuracy. eBPF is a technology that allows sandboxed programs to run within the kernel. It provides a mechanism for secure code injection during kernel and user program events, enabling developers to dynamically extend kernel functionality without modifying the kernel source code. Cgroups are a mechanism provided by the Linux kernel for allocating, limiting, isolating, and monitoring system resources such as CPU, memory, disk I / O, and network bandwidth. It allows system administrators to group processes and finely control the resource usage of each group.

[0016] Furthermore, the pre-set resource indicator thresholds in the relevant technologies are fixed thresholds that do not take into account container type and load fluctuations, resulting in low accuracy in identifying abnormal containers.

[0017] To address the aforementioned technical problems, this application provides a method and electronic device for determining the cause of container anomalies. The method includes: using a first probe to collect resource indicator values ​​of each container within a target node; identifying the abnormal container and its abnormal resource indicators when any resource indicator value of any container is abnormal, and activating a second probe; using the second probe to trace the resource access kernel call stack of the target node; identifying multiple candidate associated containers based on the resource access kernel call stack and the abnormal resource indicators; obtaining the correlation degree between each candidate associated container and the abnormal container; determining at least one target associated container and the target associated resource type corresponding to each target associated container based on the correlation degree, and activating a third probe; using the third probe, based on the target associated resource type corresponding to each target associated container, determining the target associated resource access events with call stacks for the abnormal container and each target associated container within a preset time window; and determining the resource contention chain of the abnormal container based on the target associated resource access events with call stacks, so that the user can determine the cause of the abnormal container's anomaly based on the resource contention chain. The method provided by the above solution achieves deep tracing from container abnormal resource indicators to the kernel call stack, and then to cross-container resource access events. By constructing a "resource contention link," the resource contention relationship and call path between abnormal containers and related containers are clearly revealed. This transforms anomaly cause analysis from relying on manual experience to automated and visualized diagnosis based on objective call link data, greatly improving the accuracy and efficiency of anomaly cause localization. Therefore, it can solve the technical problems of being unable to determine the cause of anomalies and the low accuracy of the obtained causes in related technologies, achieving the technical effect of improving the accuracy of the determined anomaly causes.

[0018] It should be noted that the first probe, second probe, and third probe provided in this application embodiment are implemented based on the eBPF program.

[0019] Embodiments of this application provide a method for determining the cause of container anomalies, applied to a target node, which may be a server node. Figure 1 A flowchart of the container anomaly determination method provided in the embodiments of this application is shown below. Figure 1 As shown, the method for determining the cause of the container malfunction includes: Step S101: Use the first probe to collect resource index values ​​of each container within the target node.

[0020] Resource metrics include CPU scheduling latency, Last Level Cache (LLC) miss rate, memory bandwidth utilization, and block device I / O request latency. These resource metrics can be determined using the following formula:

[0021] in, For resource indicators, CPU scheduling latency (unit: The first probe is used to mount the sched_switch kernel trace point to calculate the time taken for each task in the container to wake up (i.e., RUNNABLE state) and start running. The final cache miss rate is determined by reading the performance monitoring unit (PMU) through the first probe by attaching the performance event perf_event, and calculating the ratio of the number of LLC misses to the total number of LLC accesses in the current period. The memory bandwidth utilization rate (%) is determined by statistically analyzing the proportion of container memory read / write bandwidth to the total container bandwidth. The waiting delay (in ms) for block device input / output requests is determined by attaching the block request insertion (block_rq_insert) and block request completion (block_rq_complete) trace points to the first probe, and calculating the time difference from the initiation to the completion of the input / output (IO) request.

[0022] The first probe associates the collected kernel events, i.e. the aforementioned resource indicator values, with the Cgroup identifier (Cgroup_id) to the specific container, and outputs the resource indicator values ​​of each container at the container level.

[0023] Understandably, the target node enables the first probe by default and disables the second and third probes.

[0024] Step S102: If any resource indicator value of any container is abnormal, identify the abnormal container and its abnormal resource indicator, and activate the second probe.

[0025] Understandably, the second probe is activated when an abnormal container is present.

[0026] Step S103: Using the second probe, track the resource access kernel call stack of the target node; based on the resource access kernel call stack and abnormal resource indicators, determine multiple candidate associated containers; obtain the correlation degree between each candidate associated container and the abnormal container; based on the correlation degree, determine at least one target associated container and the target associated resource type corresponding to each target associated container, and enable the third probe.

[0027] Among them, candidate associated containers are those that may cause the abnormal container to malfunction. It is understood that the third probe is activated only if a target associated container exists. If no target associated container exists, the third probe will not be activated, and the cause of the abnormal container's anomaly will be determined manually based on the resource metric values ​​of each container.

[0028] By attaching kernel functions such as `sched_switch`, `copy_page_to_iter`, `blk_mq_submit_bio`, and `netif_rx` / `dev_queue_xmit` for network packet sending / receiving, a complete kernel-mode and user-mode call stack frame sequence is recorded when an event is triggered. This traces the kernel call stack of resource access on the target node. The call stack events are associated with specific containers using `Cgroup_id` to obtain container resource access events with call stacks. At the same time, each call stack event is tagged with a resource type based on the attached kernel function category, such as CPU contention, memory copy, I / O submission, and network transmission.

[0029] Step S104: Using the third probe, based on the target associated resource type corresponding to each target associated container, determine the target associated resource access events with call stacks for the abnormal container and each target associated container within a preset time window. Based on the target associated resource access events with call stacks, determine the resource contention chain of the abnormal container, so that the user can determine the cause of the abnormality of the abnormal container based on the resource contention chain.

[0030] The third probe is used to capture cross-container resource transfer paths. Through the "producer-consumer" link of resource access, it generates the resource contention link of the abnormal container, enabling users to determine the cause of the abnormality of the abnormal container based on the resource contention link of the abnormal container.

[0031] The preset time window is determined based on the moment when the abnormal resource indicators of the abnormal container become abnormal. Specifically, it is determined by extending forward and backward by a first duration, using the moment t0 when the abnormal resource indicators of the abnormal container become abnormal as the baseline, to define the preset time window [t0 - first duration, t0 + first duration]. It should be noted that the first duration is set by technical personnel and is not specifically limited here.

[0032] The container anomaly determination method provided in this application achieves deep tracing from container anomaly resource indicators to kernel call stacks, and then to cross-container resource access events. By constructing a "resource contention link," it clearly reveals the resource contention relationship and call path between the anomaly container and related containers, transforming anomaly cause analysis from relying on manual experience to automated and visualized diagnosis based on objective call link data, greatly improving the accuracy and efficiency of anomaly cause localization. Therefore, it can solve the technical problems of being unable to determine the anomaly cause and the low accuracy of the obtained anomaly cause in related technologies, achieving the technical effect of improving the accuracy of the determined anomaly cause.

[0033] Compared to the method of running all three probes simultaneously, the embodiments of this application adopt a hierarchical dynamic probe architecture. For example, this application configures three levels of probes. The first probe is always resident, while the second and third probes are only dynamically activated when an abnormality is detected. The three levels of probes work together to ensure the continuous collection of core resource indicators. High-overhead probes are only dynamically activated when deep tracking is required. This effectively solves the problem of high CPU overhead caused by the continuous operation of all eBPF probes, significantly reduces the overall performance loss of the system, and can meet the requirements of low latency and high performance in the production environment.

[0034] Embodiments of this application provide a method for determining the cause of container anomalies, applied to a target node, which may be a server node. Figure 2 A flowchart of the container anomaly determination method provided in the embodiments of this application is shown below. Figure 2 As shown, the method for determining the cause of the container malfunction includes: Step S201: Using the first probe, collect resource indicator values ​​for each container within the target node. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0035] Step S202: If any resource indicator value of any container is abnormal, identify the abnormal container and its abnormal resource indicator, and activate the second probe.

[0036] Specifically, step S202 includes: Step S2021: For any resource indicator value of any container, obtain the historical resource indicator value of the target resource indicator corresponding to the resource indicator value of the container.

[0037] The historical resource index values ​​corresponding to the target resource index value of the container include: resource index values ​​of the target resource index of the container collected within the same time period as the collection time of the container's resource index value within the most recent preset time period. The most recent preset time period can be the most recent 7 days. The time period that coincides with the collection time of the container's resource index value is [the collection time of the container's resource index value - the second duration, the collection time of the container's resource index value + the second duration]. The second duration can be 30 minutes, which is set by technical personnel and is not specifically limited here.

[0038] It should be noted that after obtaining the historical resource index values ​​of the target resource index corresponding to the resource index value of the container, outliers outside of 3σ are removed based on the 3σ rule. Historical resource index values ​​that deviate from the mean by more than 3σ are not included in the calculation.

[0039] Step S2022: Determine the historical baseline of the target resource indicator based on the historical resource indicator values ​​of the target resource indicator.

[0040] Based on the historical resource indicator values ​​of the target resource indicator, determine the average value of the historical resource indicator values, and use the average value of the historical resource indicator values ​​as the historical baseline of the target resource indicator for this container.

[0041] Specifically, based on the first formula, the historical baseline of the target resource indicators is determined. The first formula is:

[0042] in, This is the historical baseline value of the target resource metric for this container; The number of resource index values ​​of the target resource index of the container collected within the same time period as the collection time of the resource index value of the container in the most recent preset time period, i.e. the number of historical resource index values; W is the sliding time window, which takes the time period in the most recent 7 days that is the same time period as the collection time of the resource index value of the container. Within the most recent preset time period, the time period that coincides with the time when the resource indicator value of the container was collected. The resource index values ​​of the target resource index of the container are collected at any time.

[0043] Understandably, since historical resource indicator values ​​change over time, the historical baseline of the target resource indicator is no longer a fixed value, but rather evolves adaptively with business load and system iteration. This solves the problem in related technologies where fixed thresholds are used without considering container type and load fluctuations, resulting in low accuracy in identifying abnormal containers, and improves the accuracy of identifying abnormal containers.

[0044] In related technologies, the determination of whether a new container is abnormal is based on pre-set resource indicator thresholds. However, it's normal for a new container to experience high I / O during its initial startup. These technologies can easily misinterpret this situation, causing resource indicator values ​​to exceed pre-set thresholds, as an abnormality. Therefore, these methods are ineffective in determining whether a new container is abnormal.

[0045] Since no historical resource indicator value exists for any resource indicator of a new container, in this embodiment, if no historical resource indicator value for the target resource indicator corresponding to any resource indicator value of a container is obtained, it indicates that the container is a new container. If the number of target containers of the same type as the new container in the target cluster exceeds a preset threshold, the historical baselines of the target resource indicators of multiple target containers of the same type in the target cluster are obtained. An aggregation operation is performed on the historical baselines of the target resource indicators of multiple target containers to determine the historical baseline of the target resource indicator for the new container. For example, for a newly started order service container, all order service containers in the cluster are found, and the upper limit of 95% of normal requests with their request processing latency is obtained as the initial historical baseline of the request processing latency of the newly started order service container, avoiding interference from a few slow requests. If the number of target containers of the same type as the new container in the target cluster does not exceed a preset threshold, the historical baselines of the target resource indicators of each container in each node are obtained. Based on the historical baselines of the target resource indicators of each container in each node, the node type of each node, and the node type of the node to which the container belongs, the initial historical baseline of the target resource indicator for the new container is derived. If the initial historical baseline of the target resource indicators of the container cannot be determined through the above methods, the technical personnel shall set the initial historical baseline of the target resource indicators of the container based on industry experience and hardware specifications.

[0046] It should be noted that after the new container is launched, the system will reassess every hour whether there are enough historical resource metrics to replace the above strategy, until 7 days of historical data are accumulated and then switch to full dynamic baseline calculation.

[0047] Step S2023: Obtain the absolute value of the difference between the resource indicator value of the container and the historical baseline of the target resource indicator.

[0048] Specifically, the absolute value of the difference between the resource indicator value of the container and the historical baseline of the target resource indicator is determined by the following formula:

[0049] in, This is the absolute value of the difference between the resource indicator value of this container and the historical baseline of the target resource indicator. This is the resource metric value for the container. This serves as the historical baseline for the target resource indicators.

[0050] Step S2024: If the absolute value of the difference exceeds the difference threshold corresponding to the target resource indicator and the duration exceeds the preset duration, then the container is determined to be an abnormal container and the target resource indicator is an abnormal resource indicator.

[0051] The preset duration can be 5 seconds.

[0052] It should be noted that if the historical resource indicator value corresponding to the target resource indicator value of the container is not obtained, the determination of whether the container is an abnormal container and whether the target resource indicator is an abnormal resource indicator is based on the determined initial historical baseline of the target resource indicator of the container. Specifically, if the resource indicator value of the container is greater than the initial historical baseline of the target resource indicator of the container, the container is determined to be an abnormal container and the target resource indicator is an abnormal resource indicator.

[0053] Step S203: Using the second probe, track the resource access kernel call stack of the target node; based on the resource access kernel call stack and abnormal resource indicators, determine multiple candidate associated containers; obtain the correlation degree between each candidate associated container and the abnormal container; based on the correlation degree, determine at least one target associated container and the target associated resource type corresponding to each target associated container, and enable the third probe.

[0054] Specifically, step S203 includes: Step S2031: Determine the type of abnormal resource based on the abnormal resource indicators.

[0055] When the abnormal resource indicator is CPU scheduling latency, the abnormal resource type is determined to be CPU type; when the abnormal resource indicator is last-level cache miss rate, the abnormal resource type is determined to be cache type; when the abnormal resource indicator is memory bandwidth utilization rate, the abnormal resource type is determined to be memory type; when the abnormal resource indicator is block device I / O request wait latency, the abnormal resource type is determined to be disk type.

[0056] Step S2032: Based on the resource access kernel call stack, determine the multiple processes that call the kernel function corresponding to the abnormal resource type within a preset time window.

[0057] Step S2033: Determine the control group to which multiple processes belong.

[0058] Step S2034: Based on the control groups to which multiple processes belong, determine multiple candidate associated containers.

[0059] Specifically, by utilizing the Watch interface of the Kubernetes API Server, the Kubernetes metadata (Pod name, Namespace, Service, list of other containers within the Pod) of the candidate associated containers are associated based on the control group identifier of the control group to which the process belongs, and the associated containers within the Pod and associated containers within the Service are marked to clarify the location information of the candidate associated containers.

[0060] Step S204: Using a third probe, based on the target associated resource type corresponding to each target associated container, determine the target associated resource access events with call stacks for the abnormal container and each target associated container within a preset time window. Based on the target associated resource access events with call stacks, determine the resource contention chain of the abnormal container, so that the user can determine the cause of the abnormal container's anomaly based on the resource contention chain. For details, please refer to... Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0061] The container anomaly determination method provided in this application establishes an independent historical dynamic baseline for the resource indicators of each container, replacing the globally fixed threshold in related technologies. This ensures the accuracy of anomaly container determination and avoids false alarms. By introducing a duration judgment mechanism, it requires that an abnormal state last for more than a preset duration before triggering, thus ensuring the accuracy of anomaly container determination.

[0062] By establishing an initial baseline for the resource metrics of a new container, the cold start blind spot can be effectively eliminated, ensuring that the new container has a reasonable initial baseline and accurately determining whether the new container is abnormal.

[0063] Traditional monitoring tools can only observe abnormal resource metrics at the node or container level, but cannot answer the question of "which specific process in which container performed what operation in the kernel that caused the abnormality." This application's embodiment captures underlying kernel events through the resource access kernel call stack and, combined with the cgroup control group mechanism, achieves for the first time an end-to-end association mapping from kernel function calls to specific container instances. This allows operations personnel to directly see "which processes in which containers are competing for critical resources in the kernel at the time of the abnormality," solving the technical challenge of the observation gap between the kernel layer and the container layer in traditional monitoring.

[0064] In some optional implementations, the above-mentioned method for determining the cause of container anomalies further includes: Step a1: Determine the standard deviation of the target resource indicator based on its historical resource indicator values ​​and historical baseline.

[0065] Specifically, based on the second formula, the standard deviation of the target resource indicator is determined. The second formula is:

[0066] in, The standard deviation of the target resource indicator.

[0067] Step a2: Determine the difference threshold corresponding to the target resource indicator based on the standard deviation of the target resource indicator.

[0068] Specifically, based on the third formula, the difference threshold corresponding to the target resource indicator is determined. The third formula is:

[0069] in, The threshold value for the difference between the target resource indicators. As the pre-trigger threshold, The standard deviation of the target resource indicator.

[0070] The logic for choosing the value of k is as follows: First, most resource indicators approximately follow a normal distribution under stable operating conditions, with a 95.4% probability of data falling within μ±2σ and a 99.7% probability of data falling within μ±3σ. Second, for indicators with small fluctuations and high performance sensitivity, anomalies are strictly captured, with a tolerance of 4.6% false alarms, so k=2 is chosen. For indicators with large fluctuations and high susceptibility to external interference, false alarms are reduced, and only significant anomalies are captured, so k=3 is chosen.

[0071] Based on the characteristics of the four resource indicators in this application embodiment, CPU scheduling latency has low volatility but is sensitive to latency; LLC miss rate is greatly affected by memory access patterns and fluctuates significantly; memory bandwidth utilization is affected by container memory-intensive operations, and a trade-off can be made between sensitivity and false alarms; block device I / O request waiting latency is greatly affected by disk I / O queues and storage media, and fluctuates drastically. The default value of k is: =2, =3, =2.5, =3. That is, when the target resource metric is CPU scheduling latency, k is 2 used to determine the corresponding difference threshold; when the target resource metric is LLC missing rate, k is 3 used to determine the corresponding difference threshold; when the target resource metric is memory bandwidth utilization, k is 2.5 used to determine the corresponding difference threshold; and when the target resource metric is block device I / O request wait latency, k is 3 used to determine the corresponding difference threshold.

[0072] The container anomaly cause determination method provided in this application embodiment implements an adaptive dynamic difference threshold, replacing the rigid fixed threshold, and significantly improving the accuracy of anomaly detection and business adaptability.

[0073] In some optional implementations, step S203 above includes: Step b1: For any candidate associated container, based on the resource index values ​​of the candidate associated container and the abnormal container within the preset time window, determine the standard deviation of each resource index of the candidate associated container, the standard deviation of each resource index of the abnormal container, and the covariance of each resource index of the candidate associated container and the corresponding resource index of the abnormal container.

[0074] Step b2: Based on the standard deviation of each resource indicator of the candidate associated container, the standard deviation of each resource indicator of the abnormal container, and the covariance of each resource indicator of the candidate associated container and the corresponding resource indicator of the abnormal container, determine the correlation coefficient between each resource indicator of the candidate associated container and the corresponding resource indicator of the abnormal container.

[0075] Step b3: Based on the correlation coefficients between the resource indicators of the candidate associated container and the corresponding resource indicators of the abnormal container, determine the maximum correlation coefficient as the degree of association between the candidate associated container and the abnormal container.

[0076] It should be noted that, for high-frequency resource contention scenarios, the second probe tracks the kernel function mount points for process call latency, cross-process / container memory copying, network packet sending / receiving, and disk I / O request initiation and write operations. When these kernel functions are called, the second probe can identify the process calling the kernel function, determine the specific container based on the control group to which the process belongs, and ultimately obtain a container resource access event with a call stack. Simultaneously, based on the mounted kernel functions, the container resource access event with the call stack can be tagged with a resource type label, identifying the type of resource, such as memory copy, I / O submission, network packet sending, etc., and associated with it by combining container metadata obtained from the Kubernetes interface. The call frequency of each container and each resource type within a preset time window is statistically analyzed using an eBPF hash table. Containers with resource operation frequencies higher than the baseline frequency are marked as high-frequency operation containers. High-frequency operation containers are more likely to be associated objects of resource contention; therefore, they are prioritized as candidate associated containers, and their target associated containers are subsequently determined based on the correlation strength. In other words, in scenarios with high resource contention, the sources of candidate associated containers can include two: one is to use a second probe to track the resource access kernel call stack of the target node, and to identify multiple candidate associated containers based on the resource access kernel call stack and abnormal resource indicators; the other is to identify high-frequency operation containers as candidate associated containers.

[0077] In some alternative implementations, step b2 above includes: Step b21: Based on the fourth formula, determine the correlation coefficients between each resource indicator of the candidate associated container and the corresponding resource indicators of the abnormal container. The fourth formula is:

[0078] in, The correlation coefficient (i.e., Pearson correlation coefficient) is the correlation coefficient between the i-th resource indicator of the candidate associated container and the corresponding resource indicator of the abnormal container. Let be the standard deviation of the i-th resource metric of the candidate associated container; This is the standard deviation of the resource metric corresponding to the i-th resource metric of the candidate associated container in the abnormal container; Let be the covariance between the i-th resource metric of the candidate associated container and the corresponding resource metric of the abnormal container.

[0079] In some optional implementations, step S203 above includes: Step c1: For any candidate associated container, if the degree of association between the candidate associated container and the abnormal container is greater than the preset degree of association threshold, then the candidate associated container is determined as the target associated container, and the resource type of the resource indicator corresponding to the degree of association between the candidate associated container and the abnormal container is determined as the target associated resource type corresponding to the candidate associated container.

[0080] The preset correlation threshold can be 0.7, without specific restrictions. The resource indicator corresponding to the correlation between the candidate container and the abnormal container is the resource indicator with the highest correlation coefficient between the candidate container and the abnormal container. In resource competition scenarios, a Pearson correlation coefficient of 0.7 corresponds to the criteria for strong correlation (correlation classification: greater than 0.9 is extremely strong; greater than 0.7 and not greater than 0.9 is strong; greater than 0.4 and not greater than 0.7 is moderate). Taking 0.7 as the threshold for triggering the third probe can effectively filter out occasional noise from weak correlations and retain statistically significant competitive associations.

[0081] The container anomaly determination method provided in this application improves the accuracy of the obtained anomaly causes by using a preset correlation threshold (such as 0.7) to filter out accidental time synchronization with statistical significance, ensuring that only strongly related containers are used as target associated containers.

[0082] In some optional implementations, step S204 above includes: Step d1: Based on the access events of the abnormal container and each target associated container with call stack within a preset time window, determine the associated events of the abnormal container and each target associated container.

[0083] It should be noted that, based on the kernel call stack of the target node's resource access, multiple target processes that call various kernel functions within a preset time window are identified, and resource access events with call stacks are determined based on the control group to which the multiple target processes belong.

[0084] Based on the target associated resource type corresponding to each target associated container, the target associated resource access events with call stacks for the exception container and each target associated container within a preset time window are determined from the resource access events of the call stack.

[0085] Based on the access events of the exception container and each target associated container with call stacks within a preset time window, determine the resource type, node information, and millisecond-level timestamp in the access events of the exception container with call stacks within the preset event window. Obtain the resource type, node information, and millisecond-level timestamp in the access events of each target associated container with call stacks within the preset time window.

[0086] Based on the resource type, node information, and millisecond-level timestamp in the target associated resource access event with call stack of the abnormal container within the preset event window, and the resource type, node information, and millisecond-level timestamp in the target associated resource access event with call stack of each target associated container within the preset time window, the associated events of the abnormal container and each target associated container are determined.

[0087] For example, if the resource type, node information, and millisecond-level timestamp in the first event of the abnormal container are the same as those in the second event of the target associated container, then the first event of the abnormal container and the second event of the target associated container are determined to be associated events. The node information can be the node IP.

[0088] After the second probe determines the target associated container and the target associated resource type corresponding to each target associated container, it activates the third probe and sends information such as the identifier of the target associated container, the identifier of the abnormal container, the target associated resource type, the preset time window, the degree of association between the target associated container and the abnormal container, the call stack summary, and the resource instance ID to the third probe.

[0089] A second probe is attached at the entry / exit point of the corresponding kernel function to capture all resource access events belonging to the target associated container and abnormal containers within a preset time window. These events include IO request enqueuing (blk_mq_submit_bio), IO request completion (blk_rq_complete), sched scheduling switching (sched_switch), and memory copying (copy_page_to_iter). Each event is recorded with the following information: container ID, resource type, operation type, timestamp, call stack frame, and resource instance ID. The event information captured at the kernel level by the eBPF probe is combined with different resource types to convert the information into resource instance IDs with specific comparative meanings. For example, the device number (major, minor) in the kernel bio or request structure of a block device is generated in block:major:minor format. In Linux systems, a block device with a major device number of 8 and a minor device number of 0, represented by block:8:0, is / dev / sda.

[0090] Sort the above event sequence by time to identify the producer-consumer chain pattern: such as the queuing time of IO requests in container A. The start time of IO wait for container B exist If the causal relationship between the two is found within the same IO queue, then the cross-container resource delivery path is confirmed. This means determining the associated events between the abnormal container and each target associated container.

[0091] Step d2: Based on the association events of the abnormal container and each target associated container, generate the link identifier of at least one resource contention link of the abnormal container.

[0092] For any target associated container, determine the resource type, node information, and millisecond-level timestamp of the association event between the abnormal container and the target associated container; obtain a preset random number; and generate a link identifier for a resource contention link of the abnormal container based on the resource type, node information, millisecond-level timestamp, and preset random number of the association event between the abnormal container and the target associated container.

[0093] Specifically, the link identifier is generated using the following formula:

[0094] in, For link identification, For the preset hash function, The resource type for the associated events of the abnormal container and the target associated container. For node information, Millisecond-level timestamps This is a preset number of random numbers. The preset number of random numbers can be 8 bits.

[0095] By establishing link identifiers, resource access events scattered across different probes, containers, and times are aggregated under the same competing link, supporting end-to-end tracing and root cause analysis. In end-to-end tracing, the resource type and its instance identifier are clearly defined first to avoid confusion between resources of different types / instances.

[0096] It should be noted that when generating the link identifier for the current resource-contention link, existing link identifiers are searched. If an existing link identifier contains a target link identifier of the same resource type as the current resource-contention link and whose timestamp interval with the current resource-contention link does not exceed the third time interval, then that target link identifier is reused as the link identifier for the current resource-contention link. If no existing link identifier exists, the link identifier for the current resource-contention link is generated according to the link identifier generation formula described above.

[0097] Step d3: Using the link identifier, bind the corresponding associated event to generate the resource contention link for the abnormal container.

[0098] The associated event used to generate the link identifier is the associated event corresponding to the link identifier.

[0099] After generating a resource contention link for an abnormal container, the third and second probes are automatically shut down, while the first probe is retained.

[0100] The resource contention chain includes: chain identifier, exception container, target association container, target resource type, contention timestamp, association degree, and a complete event sequence including the call stack. This is provided for automated root cause analysis by upper-layer analysis systems.

[0101] In some optional implementations, the above-mentioned method for determining the cause of container anomalies further includes: Step e1: Determine the load value of the target node based on the processor utilization, memory utilization, and probe overhead value of the target node.

[0102] Specifically, the load value of the target node is determined based on the fifth formula, which is:

[0103] in, The load value of the target node; This refers to the processor utilization of the target node, i.e., CPU utilization. This refers to the memory utilization rate of the target node. This represents the probe overhead value, which is the CPU overhead value occupied by all initiated probes. , and As a weighting factor, + + =1, default =0.5, =0.3, =0.2.

[0104] The probe overhead value can be calculated based on formula six, which is:

[0105] in, The number of processor cycles used by the enabled probe. This represents the total number of processor cycles.

[0106] It should be noted that the target node's CPU utilization, memory usage, probe overhead values, etc., are collected by reading the / proc / stat and / proc / meminfo files and the eBPF program's built-in counters. The eBPF program records the number of CPU cycles it consumes each time it is called by the kernel through its built-in perf_event counter, and the user-space controller periodically reads and calculates the probe overhead values.

[0107] Step e2: If the load value of the target node exceeds the preset load threshold and the second probe and / or the third probe is enabled, disable the second probe and / or the third probe.

[0108] The preset load threshold can be 80%.

[0109] In some optional implementations, the above-mentioned method for determining the cause of container anomalies further includes: If the probe overhead value is greater than the preset overhead value threshold and the second probe and / or the third probe is enabled, disable the second probe and / or the third probe.

[0110] The preset overhead threshold can be 5%. Specifically, when shutting down the second probe and / or the third probe, the third probe is shut down first, followed by the second probe.

[0111] The second and third probes can only be reactivated after the probe overhead value recovers to below 5%.

[0112] It should be noted that when the probe overhead value is in the warning range of [3%, 5%], the second and third probes enable sampling mode: only one processing is performed for every N kernel events (default N=2, the higher the overhead, the larger N becomes), that is, only one of the N abnormal resource metric events is processed. This reduces overhead without completely unloading the probes and avoids the management overhead introduced by frequent mounting / unmounting.

[0113] The container anomaly determination method provided in this application implements self-protection and resource constraints for container monitoring, ensuring that monitoring tools do not become new performance problems and prioritizing business operations.

[0114] In some optional implementations, the above-mentioned method for determining the cause of container anomalies further includes: Step f1: After the user determines the cause of the abnormal container's anomaly and takes the target remediation measures, monitor the status of the abnormal container. If the abnormal container's status remains abnormal, report an alarm message indicating an incorrect location of the abnormal container's cause, so that the user can re-determine the cause of the abnormal container's anomaly based on the resource contention link of the abnormal container.

[0115] The container anomaly determination method provided in this application automatically and continuously monitors the recovery status of the abnormal container after the user implements remedial measures, forming a complete "monitoring → diagnosis → repair → verification" operation and maintenance closed loop. When it is detected that the container status has not returned to normal after repair, the system automatically reports an alarm indicating an incorrect anomaly location, forcibly triggering a secondary diagnostic process. This mechanism significantly improves the completeness and reliability of operation and maintenance response, transforming traditional passive operation and maintenance that relies on manual verification into proactive operation and maintenance with automated verification.

[0116] To make the container anomaly determination method provided in this application embodiment clearer, a specific embodiment is used for illustration. This specific embodiment describes CPU scheduling contention (high-load container causing scheduling delay of neighboring containers). Scenario description: On the same Kubernetes node, container A (machine learning inference service, high CPU usage) and container B (order processing service, latency sensitive) coexist. Container A intensively seizes CPU time slices during peak inference times, causing the CPU scheduling delay of container B to increase. Monitoring process: (1) First probe detection: The first probe collects the CPU time slices of container B every second. It was detected that its CPU scheduling latency deviated from the baseline for more than 5 consecutive seconds. (i.e., the scheduling delay increases from the baseline of 50μs to over 200μs), triggering the second probe. (2) Second probe association: The second probe attaches the sched_switch probe to track scheduling switching events and finds that the process of container A frequently preempts the CPU, causing the process of container B to be in the RUNNABLE state for a long time. Calculate the time difference between container A and container B. The correlation coefficient Corr(A, B) = 0.85 > 0.7, triggering the third probe. (3) Third probe tracking: The third probe captures all scheduling switching events within the time window, restores the complete scheduling competition link of "container A process running → container B process waiting → container B process recovering", generates ChainID = Hash(cpu_sched, 192.168.1.10, 1682000000123, abcd1234), and binds all related events. (4) Alarm output: The system outputs a structured competition link report: Noise neighbor (target associated container) = container A (ml-inference), victim container (abnormal container) = container B (order-service), resource type = CPU scheduling, correlation coefficient = 0.85, duration = 180s, ChainID = cpu_sched-192.168.1.10-....

[0117] Embodiments of this application provide a container anomaly cause determination system, applied to a target node, for implementing the container anomaly cause determination method provided in any of the above embodiments. Figure 3The container anomaly cause determination system provided in the embodiments of this application, such as Figure 3 As shown, the container anomaly determination system includes a probe management module, an overhead control module, a layered probe execution module, a resource link marking module, and a data storage module. The probe management module coordinates the layered scheduling, status monitoring, and policy decisions for the first, second, and third probes. The probe management module includes a status awareness unit, a layered decision-making unit, and a policy configuration unit. The status awareness unit collects node load metrics in real time, including CPU utilization, memory usage, and eBPF probe overhead values. It collects these node load metrics from the file system via an eBPF program and connects to the Kubernetes API Server to monitor container / Pod creation, destruction, scaling lifecycle time, and resource quota change events to promptly allocate probe resources to new containers and reclaim probe bindings for destroyed containers. The layered decision-making unit sets the probe trigger conditions: by default, the first probe is enabled, and the second and third probes are disabled. The trigger condition for the second probe is: the first probe detects an anomaly in any resource metric value of any container for more than 5 seconds. The triggering condition for the third probe: The second probe tracks cross-container resource dependency clues, such as an abnormally strong correlation between the kernel call stack of container A and the IO latency of container B, which, after preliminary correlation analysis, confirms a risk of resource contention. Degradation conditions: If the node's load value exceeds a preset load threshold or the eBPF probe overhead value exceeds a preset overhead threshold, the second and third probes are automatically shut down, leaving only the first probe. See the aforementioned descriptions for details, which will not be repeated here. The policy configuration unit supports user-defined tiered thresholds, resource type priorities, correlation thresholds, system load degradation thresholds, time-based container parameters, etc.

[0118] The hierarchical probe execution module is used to execute eBPF monitoring logic of corresponding granularity according to the trigger conditions of the probes set by the hierarchical decision unit, and generate raw resource access data. The hierarchical probe execution module includes a first probe execution unit, a second probe execution unit, and a third probe execution unit. The first probe execution unit controls the operation of the first probe; its purpose is described above and will not be repeated here. The second probe execution unit controls the operation of the second probe; its purpose is described above and will not be repeated here. The third probe execution unit controls the operation of the third probe. The third probe monitors cross-container resource transfer paths, generating resource access chain IDs through the "producer-consumer" link of resource access. The access chain format is: resource type-node IP-timestamp-random number. For example, if an IO request from container A is queued → a request from container B is queued, the ID is io_block-192.168.1.10-1620000000-abc123. This ID is shared within the same competing link. The purpose of the third probe is described above and will not be repeated here.

[0119] The resource link tagging module is used to uniquely identify cross-container resource contention links, supports full-link association, and provides judgment and traceability for deep-layer full-link tracing. ChainID association is completed through "clue triggering → event aggregation → ID allocation → full-link binding". For details, please refer to the relevant descriptions above, which will not be repeated here.

[0120] The data storage module is used to store cross-container resource contention links.

[0121] The overhead control module includes a dynamic unloading and mounting unit. This unit automatically mounts a second probe when the node load is normal and the first probe detects an abnormal container; it also automatically mounts a third probe when the second probe finds a strong correlation clue. When the node load exceeds the limit, probes are unloaded in the order of the third probe, the second probe, and the first probe. When a container is destroyed, its associated probes are automatically cleaned up.

[0122] The overhead control module also includes a sampling adjustment unit, which is used to enable the sampling modes of the second and third probes when the probe overhead value is in the [3%, 5%] warning range. For details, please refer to the corresponding description above, which will not be repeated here.

[0123] Figure 4 This is a schematic diagram of the state transition of the layered probe provided in an embodiment of this application, as shown below. Figure 4 As shown, the first probe is enabled by default. When any resource metric value of any container is abnormal, the second probe is enabled. If the second probe determines that a target associated container exists, the third probe is enabled. If the second probe determines that no target associated container exists, the second probe is disabled, and the system reverts to the state where the first probe is enabled. When resource contention is resolved, the third probe is disabled, and the system reverts to the state where the second probe is enabled. If the node load and the eBPF probe cost value exceed the corresponding threshold, the second and third probes are disabled, and the system degrades to a degraded state, where only the first probe is enabled. If the node load and the eBPF probe cost value do not exceed the corresponding threshold, the degraded state can jump back to the state where the first probe is enabled.

[0124] The container anomaly determination system provided in this application employs a layered dynamic probe architecture. For example, this application configures a three-level probe system. The first probe is always resident, while the second and third probes are dynamically activated only when anomalies are detected. The three-level probes work collaboratively to ensure continuous collection of core resource metrics, dynamically activating high-overhead probes only when deep tracing is needed. This effectively solves the high CPU overhead problem caused by the continuous operation of all eBPF probes, significantly reducing overall system performance loss and meeting the low latency and high performance requirements of production environments. By tracing the resource access kernel call stack of the target node, and based on the resource access kernel call stack and abnormal resource metrics, a set of candidate associated containers is determined. Then, the correlation degree between each candidate associated container and the abnormal container is determined, and based on the correlation degree, at least one target association and the target associated resource type corresponding to each target associated container are determined. Finally, the resource contention chain of the abnormal container is determined. This solution surpasses traditional single-container / fixed-metric monitoring, achieving deep tracing from container resource anomaly metrics to kernel call stacks, and then to cross-container resource access events. By constructing a "resource contention link," the resource contention relationship and call path between abnormal containers and related containers are clearly revealed. This transforms root cause analysis from relying on human experience to automated and visualized diagnosis based on objective call link data, greatly improving the accuracy and efficiency of problem localization.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0126] Embodiments of this application also provide a device for determining the cause of container malfunctions, such as... Figure 5 As shown, the container anomaly cause determination device includes: The acquisition module 501 is used to acquire resource index values ​​of each container within the target node using the first probe.

[0127] The first determination module 502 is used to determine the abnormal container and its abnormal resource indicators when any resource indicator value of any container is abnormal, and to enable the second probe.

[0128] The second determining module 503 is used to use the second probe to track the resource access kernel call stack of the target node; determine multiple candidate associated containers based on the resource access kernel call stack and abnormal resource indicators; obtain the degree of association between each candidate associated container and the abnormal container; determine at least one target associated container and the target associated resource type corresponding to each target associated container based on the degree of association, and enable the third probe.

[0129] The third determination module 504 is used to use a third probe to determine the access events of the abnormal container and each target associated container with call stacks within a preset time window based on the target associated resource type corresponding to each target associated container. Based on the access events of the target associated resources with call stacks, the resource contention chain of the abnormal container is determined so that the user can determine the cause of the abnormality of the abnormal container based on the resource contention chain.

[0130] In an optional implementation, the first determining module 502 includes: The first acquisition unit is used to acquire the historical resource index value of the target resource index corresponding to any resource index value of any container.

[0131] The first determining unit is used to determine the historical baseline of the target resource indicator based on the historical resource indicator values ​​of the target resource indicator.

[0132] The second acquisition unit is used to acquire the absolute value of the difference between the resource indicator value of the container and the historical baseline of the target resource indicator.

[0133] The second determining unit is used to determine that if the absolute value of the difference exceeds the difference threshold corresponding to the target resource indicator and the duration exceeds the preset duration, then the container is an abnormal container and the target resource indicator is an abnormal resource indicator.

[0134] In an optional implementation, the first determining unit includes: The first determining subunit is used to determine the historical baseline of the target resource indicators based on the first formula, which is:

[0135] in, The historical baseline value of the target resource indicator; This refers to the number of resource indicator values ​​of the target resource indicator of the container collected within the same time period as the time when the resource indicator value of the container was collected in the most recent preset time period. Within the most recent preset time period, the time period that coincides with the time when the resource indicator value of the container was collected. The resource index values ​​of the target resource index of the container are collected at any time.

[0136] In an optional implementation, the container anomaly cause determination device further includes: The third determining unit is used to determine the standard deviation of the target resource indicator based on the historical resource indicator values ​​and historical baseline.

[0137] The fourth determining unit is used to determine the difference threshold corresponding to the target resource indicator based on the standard deviation of the target resource indicator.

[0138] In an optional implementation, the third determining unit includes: The third determining sub-unit is used to determine the standard deviation of the target resource index based on the second formula, which is:

[0139] in, The standard deviation of the target resource indicator.

[0140] In an optional implementation, the fourth determining unit includes: The fourth determining subunit is used to determine the difference threshold corresponding to the target resource indicator based on the third formula, which is:

[0141] in, The threshold value for the difference between the target resource indicators. As the pre-trigger threshold, The standard deviation of the target resource indicator.

[0142] In an optional implementation, the second determining module 503 includes: The fifth determination unit is used to determine the type of abnormal resource based on abnormal resource indicators.

[0143] The sixth determining unit is used to determine multiple processes that call the kernel function corresponding to the abnormal resource type within a preset time window based on resource access to the kernel call stack.

[0144] The seventh determining unit is used to determine the control group to which multiple processes belong.

[0145] The eighth determining unit is used to determine multiple candidate associated containers based on the control groups to which multiple processes belong.

[0146] In an optional implementation, the second determining module 503 includes: The ninth determining unit is used to determine, for any candidate associated container, the standard deviation of each resource indicator of the candidate associated container, the standard deviation of each resource indicator of the abnormal container, and the covariance of each resource indicator of the candidate associated container and the corresponding resource indicator of the abnormal container within a preset time window, based on the resource indicator values ​​of the candidate associated container and the abnormal container within a preset time window.

[0147] The tenth determining unit is used to determine the correlation coefficient between each resource indicator of the candidate associated container and the corresponding resource indicator of the abnormal container based on the standard deviation of each resource indicator of the candidate associated container, the standard deviation of each resource indicator of the abnormal container, and the covariance between each resource indicator of the candidate associated container and the corresponding resource indicator of the abnormal container.

[0148] The eleventh determining unit is used to determine the maximum correlation coefficient as the degree of association between the candidate associated container and the abnormal container based on the correlation coefficients between the various resource indicators of the candidate associated container and the corresponding resource indicators of the abnormal container.

[0149] In an optional implementation, the tenth determining unit includes: The tenth determining subunit is used to determine the correlation coefficients between the resource indicators of the candidate associated container and the corresponding resource indicators of the abnormal container based on the fourth formula, which is:

[0150] in, Let be the correlation coefficient between the i-th resource indicator of the candidate associated container and the corresponding resource indicator of the abnormal container; Let be the standard deviation of the i-th resource metric of the candidate associated container; This is the standard deviation of the resource metric corresponding to the i-th resource metric of the candidate associated container in the abnormal container; Let be the covariance between the i-th resource metric of the candidate associated container and the corresponding resource metric of the abnormal container.

[0151] In an optional implementation, the second determining module 503 includes: The twelfth determining unit is used to determine, for any candidate associated container, if the degree of association between the candidate associated container and the abnormal container is greater than a preset degree of association threshold, that the candidate associated container is determined as the target associated container, and the resource type of the resource indicator corresponding to the degree of association between the candidate associated container and the abnormal container is determined as the target associated resource type corresponding to the candidate associated container.

[0152] In an optional implementation, the third determining module 504 includes: The thirteenth determining unit is used to determine the associated events of the abnormal container and each target associated container based on the target associated resource access events with call stacks within a preset time window.

[0153] The first generation unit is used to generate a link identifier for at least one resource contention link of the abnormal container based on the association events of the abnormal container and each target associated container.

[0154] The second generation unit is used to generate resource contention links for the abnormal container by using the link identifier to bind the corresponding associated events.

[0155] In an optional implementation, the container anomaly cause determination device further includes: The fourteenth determining unit is used to determine the load value of the target node based on the processor utilization, memory utilization, and probe overhead value of the target node.

[0156] The shutdown unit is used to shut down the second probe and / or the third probe when the load value of the target node exceeds a preset load threshold and the second probe and / or the third probe is enabled.

[0157] In an optional implementation, the fourteenth determining unit includes: The fourteenth determination sub-unit is used to determine the load value of the target node based on the fifth formula, which is:

[0158] in, The load value of the target node; The processor utilization rate of the target node; This refers to the memory utilization rate of the target node. This represents the probe overhead value. , and As a weighting factor, + + =1.

[0159] Based on the sixth formula, the probe overhead value is determined. The sixth formula is:

[0160] in, The number of processor cycles used by the enabled probe. This represents the total number of processor cycles.

[0161] For a description of the features in the embodiment corresponding to the container anomaly cause determination device, please refer to the relevant description of the embodiment corresponding to the container anomaly cause determination method, which will not be repeated here.

[0162] This invention provides a container anomaly cause determination device. Through a layered dynamic probe architecture, such as the three-level probe configuration in this application, the first probe is always resident, while the second and third probes are dynamically activated only when anomalies are detected. The three-level probes work collaboratively to ensure continuous collection of core resource indicators, dynamically activating high-overhead probes only when deep tracing is needed. This effectively solves the high CPU overhead problem caused by the continuous operation of all eBPF probes, significantly reducing overall system performance loss and meeting the low latency and high performance requirements of production environments. By tracking the resource access kernel call stack of the target node, and based on the resource access kernel call stack and abnormal resource indicators, a set of candidate associated containers is determined. Then, the correlation degree between each candidate associated container and the abnormal container in the associated container set is determined, and based on the correlation degree, at least one target association and the target associated resource type corresponding to each target associated container are determined. Finally, the resource contention link of the abnormal container is determined. This solution surpasses traditional single-container / fixed indicator monitoring, achieving deep tracing from container resource anomaly indicators to the kernel call stack, and then to cross-container resource access events. By constructing a "resource contention link," the resource contention relationship and call path between abnormal containers and related containers are clearly revealed. This transforms root cause analysis from relying on human experience to automated and visualized diagnosis based on objective call link data, greatly improving the accuracy and efficiency of problem localization.

[0163] Embodiments of this application also provide an electronic device, such as... Figure 6 As shown, it includes a processor 601 and a memory 602, in which a computer program is stored. The processor 601 is configured to run the computer program to perform the steps in any of the above embodiments of the container anomaly cause determination method.

[0164] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the container exception cause determination method when it runs.

[0165] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0166] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the container exception cause determination method.

[0167] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the container exception cause determination method.

[0168] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0169] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] The above provides a detailed description of the method for determining the cause of container anomalies and the electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for determining the cause of container anomalies, characterized in that, include: The first probe is used to collect resource index values ​​of each container within the target node; If any resource indicator value of any container is abnormal, identify the abnormal container and its abnormal resource indicators, and activate the second probe. Using the second probe, the resource access kernel call stack of the target node is traced; based on the resource access kernel call stack and the abnormal resource indicators, multiple candidate associated containers are identified; and the correlation degree between each candidate associated container and the abnormal container is obtained. Based on the correlation, at least one target associated container and the target associated resource type corresponding to each target associated container are determined, and a third probe is enabled; Using the third probe, based on the target associated resource type corresponding to each target associated container, the target associated resource access events with call stacks for the abnormal container and each target associated container within a preset time window are determined. Based on the target associated resource access events with call stacks, the resource contention chain of the abnormal container is determined, so that the user can determine the cause of the abnormality of the abnormal container based on the resource contention chain.

2. The method according to claim 1, characterized in that, The process of identifying the abnormal container and its abnormal resource indicators when any resource indicator value of any container is abnormal includes: For any resource indicator value of any container, obtain the historical resource indicator value of the target resource indicator corresponding to that resource indicator value of the container; Based on the historical resource indicator values ​​of the target resource indicator, determine the historical baseline of the target resource indicator; Obtain the absolute value of the difference between the resource indicator value of the container and the historical baseline of the target resource indicator; If the absolute value of the difference exceeds the difference threshold corresponding to the target resource indicator, and the duration exceeds the preset duration, then the container is determined to be an abnormal container, and the target resource indicator is an abnormal resource indicator.

3. The method according to claim 2, characterized in that, The historical resource index values ​​of the target resource index include the resource index values ​​of the target resource index of the container collected within the same time period as the time when the resource index value of the container was collected in the most recent preset time period. Determining the historical baseline of the target resource indicator based on its historical values ​​includes: Based on the first formula, the historical baseline of the target resource indicator is determined. The first formula is: in, The historical baseline value of the target resource indicator; This refers to the number of resource indicator values ​​of the target resource indicator of the container collected within the same time period as the time when the resource indicator value of the container was collected in the most recent preset time period. Within the most recent preset time period, the time period that coincides with the time when the resource indicator value of the container was collected. The resource index values ​​of the target resource index of the container are collected at any time.

4. The method according to claim 3, characterized in that, The method further includes: Based on the historical resource indicator values ​​and historical baseline of the target resource indicator, determine the standard deviation of the target resource indicator; Based on the standard deviation of the target resource indicator, the difference threshold corresponding to the target resource indicator is determined.

5. The method according to claim 4, characterized in that, The step of determining the standard deviation of the target resource indicator based on its historical resource indicator values ​​and historical baseline includes: Based on the second formula, the standard deviation of the target resource indicator is determined. The second formula is: in, denoted as the standard deviation of the target resource indicator.

6. The method according to claim 4, characterized in that, Based on the standard deviation of the target resource indicator, determine the difference threshold corresponding to the target resource indicator, including: Based on the third formula, the difference threshold corresponding to the target resource indicator is determined, and the third formula is: in, The difference threshold corresponding to the target resource indicator. As the pre-trigger threshold, denoted as the standard deviation of the target resource indicator.

7. The method according to claim 1, characterized in that, Based on the resource access kernel call stack and the abnormal resource indicators, multiple candidate associated containers are determined, including: Based on the aforementioned abnormal resource indicators, the type of abnormal resource is determined; Based on the resource access kernel call stack, multiple processes that call the kernel function corresponding to the abnormal resource type within a preset time window are identified. Determine the control group to which the multiple processes belong; The multiple candidate associated containers are determined based on the control groups to which the multiple processes belong.

8. The method according to claim 1, characterized in that, The step of obtaining the association degree between each of the candidate associated containers and the abnormal container includes: For any candidate associated container, based on the resource index values ​​of the candidate associated container within a preset time window and the resource index values ​​of the abnormal container within the preset time window, determine the standard deviation of each resource index of the candidate associated container, the standard deviation of each resource index of the abnormal container, and the covariance between each resource index of the candidate associated container and the corresponding resource index of the abnormal container. Based on the standard deviation of each resource indicator of the candidate associated container, the standard deviation of each resource indicator of the abnormal container, and the covariance between each resource indicator of the candidate associated container and the corresponding resource indicator of the abnormal container, the correlation coefficient between each resource indicator of the candidate associated container and the corresponding resource indicator of the abnormal container is determined. Based on the correlation coefficients between the resource indicators of the candidate associated container and the corresponding resource indicators of the abnormal container, the maximum correlation coefficient is determined as the degree of association between the candidate associated container and the abnormal container.

9. The method according to claim 8, characterized in that, The determination of the correlation coefficient between each resource indicator of the candidate associated container and the corresponding resource indicator of the abnormal container, based on the standard deviation of each resource indicator of the candidate associated container, the standard deviation of each resource indicator of the abnormal container, and the covariance between each resource indicator of the candidate associated container and the corresponding resource indicator of the abnormal container, includes: Based on the fourth formula, the correlation coefficients between each resource indicator of the candidate associated container and the corresponding resource indicator of the abnormal container are determined. The fourth formula is: in, Let be the correlation coefficient between the i-th resource indicator of the candidate associated container and the corresponding resource indicator of the abnormal container; Let be the standard deviation of the i-th resource metric of the candidate associated container; This is the standard deviation of the resource metric corresponding to the i-th resource metric of the candidate associated container in the abnormal container; Let be the covariance between the i-th resource metric of the candidate associated container and the corresponding resource metric of the abnormal container.

10. The method according to claim 1, characterized in that, The step of determining at least one target associated container and the target associated resource type corresponding to each target associated container based on the association degree includes: For any candidate associated container, if the degree of association between the candidate associated container and the abnormal container is greater than a preset degree of association threshold, then the candidate associated container is determined as the target associated container, and the resource type of the resource indicator corresponding to the degree of association between the candidate associated container and the abnormal container is determined as the target associated resource type corresponding to the candidate associated container.

11. The method according to claim 1, characterized in that, The step of determining the resource contention chain of the abnormal container based on the target associated resource access event with the call stack includes: Based on the access events of the abnormal container and each of the target associated containers with call stacks within a preset time window, determine the association events of the abnormal container and each of the target associated containers. Based on the association events of the abnormal container and each of the target associated containers, generate a link identifier for at least one resource contention link of the abnormal container; Using the link identifier, the corresponding associated event is bound to generate the resource contention link of the abnormal container.

12. The method according to claim 1, characterized in that, The method further includes: The load value of the target node is determined based on the processor utilization, memory utilization, and probe overhead value of the target node. If the load value of the target node exceeds a preset load threshold, and the second probe and / or the third probe is enabled, then the second probe and / or the third probe shall be disabled.

13. The method according to claim 12, characterized in that, The determination of the load value of the target node based on the processor utilization, memory utilization, and probe overhead value of the target node includes: Based on the fifth formula, the load value of the target node is determined. The fifth formula is: in, The load value of the target node; The processor utilization rate of the target node; The memory utilization rate of the target node; The probe overhead value; , and As a weighting factor, + + =1; The probe overhead value is determined based on the sixth formula, which is: in, The number of processor cycles used by the enabled probe. This represents the total number of processor cycles.

14. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the container anomaly cause determination method as described in any one of claims 1 to 13 when executing the computer program.

Images