A fault monitoring system and method for a k8s cluster

Abstract

Embodiments of the present application provide a kind of K8S cluster fault monitoring system and method, system includes: K8S user end, for according to target monitoring demand from the preset candidate BCC template in K8S cluster selects target BCC template, and according to target BCC template generates BCC tool CRD resource and BCC rule CRD resource;Monitoring device is deployed in each working node in K8S cluster, for pulling BCC tool CRD resource from K8S cluster, based on BCC tool CRD resource analysis obtains BCC tool, by running BCC tool is monitored to the working node, obtains monitoring result data;Monitoring result collection device is deployed in each working node in K8S cluster, for pulling BCC rule CRD resource from K8S cluster, based on BCC rule CRD resource analysis obtains monitoring rule, according to monitoring rule and the monitoring result data of working node, generates the Metrics of working node, according to monitoring rule to Metrics establish normal or abnormal mark;Analysis reporter, for receiving each Metrics of each working node, analysis received all Metrics, obtains fault analysis report.

Description

Technical Field

[0001] This invention relates to the field of Kubernetes anomaly detection, and more particularly to a fault monitoring system and method for a K8S cluster. Background Technology

[0002] During the migration of departmental projects to containers, more and more projects need to be migrated to containerized platforms. As the number of Kubernetes clusters on containerized platforms increases, the service-to-service call relationships, container running status, operating system running status, basic software running status, and so on may all lead to application anomalies.

[0003] In the process of developing this invention, the applicant discovered at least the following problems in the prior art:

[0004] Existing technologies can only provide anomaly detection at a specific level and cannot perform multi-level anomaly analysis and detection, including operating systems, application components, container environments, and network environments. The data is lengthy, complex, and not easy to use. Summary of the Invention

[0005] This invention provides a fault monitoring system and method for K8S clusters, which solves the problems of existing technologies that can only provide anomaly detection at a specific level and cannot perform multi-level anomaly analysis and detection, including operating system, application components, container environment, network environment, etc., and the data is verbose, complex, and difficult to use.

[0006] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a fault monitoring system for a K8S cluster, comprising:

[0007] The K8S client is used to select a target BCC template from the preset candidate BCC templates in the K8S cluster according to the target monitoring requirements, generate BCC tool CRD resources and BCC rule CRD resources according to the target BCC template, and submit the BCC tool CRD resources and BCC rule CRD resources to the K8S cluster.

[0008] The monitoring device is deployed on each working node in the K8S cluster. It is used to pull the BCC tool CRD resource from the K8S cluster, parse the BCC tool CRD resource to obtain the BCC tool, run the BCC tool to monitor the working node, obtain monitoring result data, and transmit the monitoring result data to the monitoring result collection device on the working node.

[0009] The monitoring result collection device is deployed on each working node in the K8S cluster. It is used to pull the BCC rule CRD resource from the K8S cluster, parse the monitoring rule based on the BCC rule CRD resource, generate the metric of the working node according to the monitoring rule and the monitoring result data of the working node, establish normal or abnormal marking of the metric according to the monitoring rule, and send it to the analysis reporter.

[0010] The analysis reportr is used to receive each metric from each working node, analyze all received metrics, and obtain a fault analysis report.

[0011] Secondly, embodiments of the present invention provide a fault monitoring method for a K8S cluster, comprising:

[0012] According to the target monitoring requirements, a target BCC template is selected from the preset candidate BCC templates in the K8S cluster, and BCC tool CRD resources and BCC rule CRD resources are generated according to the target BCC template. The BCC tool CRD resources and BCC rule CRD resources are then submitted to the K8S cluster.

[0013] Pull the BCC rule CRD resource from the K8S cluster, parse the BCC tool CRD resource to obtain the BCC tool, run the BCC tool to monitor the working node, and obtain the monitoring result data on the working node.

[0014] Pull the BCC rule CRD resource from the K8S cluster, parse the monitoring rules based on the BCC rule CRD resource, generate the metric for the working node based on the monitoring rules and the monitoring result data of the working node, and mark the metric as normal or abnormal according to the monitoring rules.

[0015] Receive each metric from each working node, analyze all received metrics, and obtain a fault analysis report.

[0016] The above technical solution has the following beneficial effects: By defining BCC resource types, including BCC rule CRD resources and BCC rule CRD resources, through the CRD function in the Kubernetes system, and deploying monitoring devices and monitoring result collection devices via DaemonSet in the Kubernetes system, each worker node automatically runs these devices. Furthermore, each worker node can access BCC rule CRD resources in the same way as standard Kubernetes resources. Combined with selecting the desired target BCC template from candidate BCC templates, multi-level monitoring of the operating system, application software, container environment, and / or network environment is achieved, enabling multi-level anomaly analysis and detection for the Kubernetes system. By obtaining monitoring rule result data from BCC rule CRD resources, automatic analysis and display of monitoring result data are achieved, avoiding the need for ordinary development, testing, and operations personnel to directly analyze lengthy data, thus reducing the difficulty of use. Ordinary technicians do not need to be familiar with BCC tools, nor do they need to know how to run BCC tools on various working nodes in the K8S system. Ordinary technicians only need to select the appropriate target BCC template according to their own needs to be proficient in applying the fault detection function of the corresponding BCC tool in the K8S system, which lowers the technical threshold for ordinary technicians. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a structural diagram of a fault monitoring system for a K8S cluster, one of the embodiments of the present invention;

[0019] Figure 2 This is a flowchart of a fault monitoring method for a K8S cluster, which is one of the embodiments of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Definitions:

[0022] Kubernetes (K8S): A container orchestration and management tool;

[0023] CRD (Custom Resource Definition): A custom resource definition interface and method provided by Kubernetes;

[0024] Kubernetes native resources, such as Deployments, Services, and Ingress, are used for deploying business applications.

[0025] A container; a container image is a lightweight, standalone executable package of software that contains everything needed to run it, such as code, runtime, system tools, system libraries, and settings. A container is the running state of a program within a container image.

[0026] Pod: The smallest unit of service in Kubernetes.

[0027] BCC: BPF Compiler Collection is a collection of tools for detecting anomalies or performance issues in operating systems, software, networks, file systems, CPUs, and memory, based on eBPF.

[0028] BPF: Berkeley Packet Filter, a raw interface of the data link layer on Unix-like systems, providing raw link layer packet sending and receiving;

[0029] eBPF: Extended Berkeley Packet Filter, is a general-purpose execution engine extension based on BPF, used for developing performance analysis tools and software-defined networking, etc.

[0030] The inventors discovered that some existing technical solutions support analyzing network relationships between containers using network components. While some independent tools offer multi-layered, unrelated data analysis and detection, these existing solutions have the following drawbacks: they can only analyze network relationships between containers and detect connectivity between containers, but cannot analyze detailed information about specific connectivity, making it impossible to pinpoint problems; they can only provide anomaly detection at a specific level, resulting in lengthy, complex, and difficult-to-use data; they are very unfriendly to non-professional users, with a high learning curve; and they do not support multi-layered anomaly analysis and detection of container environments (operating system, application components, container environment, and network environment, etc.). Based on these shortcomings, the inventors solved the following problems through the following embodiments: supporting fully automated multi-layered anomaly analysis and detection of container environments; supporting template configuration, allowing users to implement detection functions by configuring parameters and selecting templates, which is very simple and easy to use, requiring no professional knowledge; and supporting a monitoring dashboard, displaying the system's operating status on a single screen.

[0031] Firstly, such as Figure 1 As shown, this embodiment of the invention provides a fault monitoring system for a K8S cluster, comprising:

[0032] K8S client 10 is used to select a target BCC template from the preset candidate BCC templates in K8S cluster 16 according to the target monitoring requirements, generate BCC tool CRD resources and BCC rule CRD resources according to the target BCC template, and submit the BCC tool CRD resources and BCC rule CRD resources to the K8S cluster.

[0033] The monitoring device 11 is deployed on each working node in the K8S cluster 16. It is used to pull the BCC tool CRD resource from the K8S cluster 16, parse the BCC tool CRD resource to obtain the BCC tool, run the BCC tool to monitor the working node, obtain monitoring result data, and transmit the monitoring result data to the monitoring result collection device 12 on the working node.

[0034] The monitoring result collection device 12 is deployed on each working node in the K8S cluster 16. It is used to pull the BCC rule CRD resource from the K8S cluster 16, parse the monitoring rule based on the BCC rule CRD resource, generate the metric of the working node according to the monitoring rule and the monitoring result data of the working node, establish normal or abnormal marking of the metric according to the monitoring rule, and send it to the analysis reporter 13.

[0035] Analysis reporter 13 is used to receive each metric of each working node, analyze all received metrics, and obtain a fault analysis report.

[0036] In some embodiments, the monitoring device 11 and the monitoring result collection device 12 are deployed on worker nodes in a K8S cluster via a Daemonset of the K8S system; the K8S client 10 can be deployed on any node in the K8S cluster; the analysis reportr 13 is unique in the cluster and can be deployed on any node in the K8S cluster; the K8S client 10 can select one or more candidate BCC templates as the target BCC template from the candidate BCC templates to monitor one or more monitoring objects, and the candidate BCC templates include BCCs that fulfill specific monitoring requirements. The tools and monitoring rules corresponding to the monitoring requirements and BCC tools; each candidate BCC template is paired with BCC tools and monitoring rules applied to monitoring one or more monitoring objects; the monitoring objects include, but are not limited to, operating systems, application software, networks, file systems, CPUs, and memory; the BCC tool CRD resources and BCC rule CRD resources are obtained through the target BCC template and submitted to the K8S cluster, specifically to the Etcd key-value store system in the K8S cluster. Other worker nodes can pull the required BCCs from the Etcd key-value store system as if using standard K8S resources. The system includes CRD resources for tools and CRD resources for BCC rules. Each BCC tool can perform a specific monitoring function, such as monitoring anomalies or performance issues in the operating system, application software, network, file system, CPU, and memory, and obtaining monitoring result data. Monitoring rules are used to constrain and check the monitoring result data returned by the BCC tools to detect any anomalies. The monitoring result data and corresponding monitoring rules obtained on each working node are encapsulated into Metrics format for collection by the Metrics collector in Kubernetes. Based on the monitoring rules, the Metrics are marked as normal or abnormal. Finally, the Metrics data of all nodes are handed over to the analysis report 13 to generate a fault analysis report. The analysis report 13 receives the Metrics data of all nodes and analyzes the Metrics at the operating system level, the Metrics of the Kubernetes cluster software, the Metrics of the application, and the Metrics of the network based on the anomaly or normal markings of the Metrics data. It assigns a priority to each Metric and determines the probability of anomalies from high to low priority, with higher priority Metrics having a greater probability of anomalies. The analysis report sends the analysis results, i.e., the fault analysis report, to the K8S cluster through the kubelet client on the same node as the analysis report. It can also be further sent out through the K8S cluster, for example, to a preset receiving end or display end.

[0037] The embodiments of this invention have the following technical effects: By defining BCC resource types, including BCC rule CRD resources and BCC rule CRD resources, through the CRD function in the Kubernetes system, and deploying monitoring devices and monitoring result collection devices through the DaemonSet method in the Kubernetes system, each worker node automatically runs the monitoring device and monitoring result collection device. Furthermore, each worker node can acquire BCC rule CRD resources and BCC rule CRD resources in the same way as acquiring standard Kubernetes resources. Combined with selecting the required target BCC template from the candidate BCC templates, multi-level monitoring is performed on the operating system, application software, container environment, and / or network environment, realizing multi-level anomaly analysis and detection for the Kubernetes system. By obtaining monitoring rule monitoring result data from the BCC rule CRD resources, automatic analysis and display of monitoring result data are achieved, avoiding the need for ordinary development, testing, and operation and maintenance technicians to directly analyze lengthy data, thus reducing the difficulty of use. Ordinary technicians do not need to be familiar with BCC tools, nor do they need to know how to run BCC tools on various working nodes in the K8S system. Ordinary technicians only need to select the appropriate target BCC template according to their own needs to be proficient in applying the fault detection function of the corresponding BCC tool in the K8S system, which lowers the technical threshold for ordinary technicians.

[0038] Furthermore, the system also includes: a K8S control terminal 14;

[0039] The K8S control terminal 14 is used to create at least one candidate BCC template based on the received information of BCC tools and monitoring rules corresponding to specific monitoring needs, and submit it to the K8S cluster.

[0040] In some embodiments, technicians familiar with BCC tools and Kubernetes systems pre-create at least one candidate BCC template based on specific monitoring needs. Each candidate BCC template records information about the BCC tools and monitoring rules used to monitor one or more monitoring objects. The monitoring objects include, but are not limited to, operating systems, application software, networks, file systems, CPUs, and memory. For technicians unfamiliar with BCC tools, they can directly select one or more candidate BCC templates from the pre-created candidate BCC templates to configure the working nodes in Kubernetes based on the target monitoring needs, thereby enabling the application of BCC tools for performance and fault monitoring in Kubernetes.

[0041] Furthermore, the monitoring device 11 includes:

[0042] BCC client 111 is used to pull the BCC tool CRD resource from the K8S cluster, parse the BCC tool CRD resource to obtain the BCC tool corresponding to the target monitoring requirement, and submit the obtained BCC tool to the BCC tool execution device 112 on the working node; and to receive monitoring result data from the BCC tool execution device 112 on the working node, and transmit the received monitoring result data to the monitoring result collection device 12 on the working node.

[0043] BCC tool execution device 112 is used to receive BCC tools from BCC client 111 on the working node, run the BCC tools in the manner of a daemon process to monitor the working node, collect monitoring result data, and transmit the monitoring result data to BCC client 111 on the working node.

[0044] In some embodiments, the monitoring device 11 can be deployed as a BCC client 111 and a BCC tool execution device 112. The BCC client 111 is responsible for obtaining the BCC tool from the K8S cluster and obtaining the monitoring result data obtained from the execution of the BCC tool. The BCC tool execution device 112 is used to run the BCC tool as a daemon process to ensure that the BCC tool can continuously and stably monitor the operation of the worker nodes.

[0045] Furthermore, the monitoring result collection device 12 includes:

[0046] The monitoring rule acquisition device 121 is used to pull the BCC rule CRD resource from the K8S cluster through the Kubelet client program deployed on the working node, parse the monitoring rule based on the BCC rule CRD resource, and transmit the parsed monitoring rule to the Metrics data generation device 122 on the working node.

[0047] Metrics data generation device 122 is used to receive monitoring rules from monitoring rule acquisition device 121 on the working node and monitoring result data from monitoring device on the working node, and encapsulate the received monitoring rules and monitoring result data into metric data.

[0048] Metrics data collector 123 is used to collect metric data on the working node and determine whether the monitoring result data encapsulated in the metric data has triggered the threshold defined by the monitoring rule, and whether the change value of the monitoring result data obtained in the current monitoring period relative to the monitoring result data of the previous fixed period exceeds a preset threshold. If it is determined that the monitoring result data encapsulated in the metric has triggered the threshold defined by the monitoring rule or the change value exceeds the preset threshold, the metric data is marked as abnormal; otherwise, the metric data is marked as normal, and all metric data is transmitted to the analysis report 13.

[0049] In some embodiments, the monitoring rule acquisition device 121 pulls the BCC rule CRD resources from the K8S cluster through the Kubelet client program on its working node, and obtains the monitoring rules. The monitoring rules are then provided to the Metrics data generation device 122. The Metrics data generation device 122 collects the monitoring rules and monitoring result data, encapsulates the monitoring rules and monitoring result data into Metrics data according to the Metrics data protocol format, and sends the obtained Metrics data to the analysis report device 13. The analysis report device 13 parses the monitoring rules and corresponding monitoring result data in each Metrics from the Metrics data, determines whether there are any anomalies in the monitoring result data based on the monitoring rules, and generates a fault analysis report. The analysis report device 13 can obtain the Metrics data of all working nodes in the K8S cluster, so it can combine all the Metrics data for analysis to obtain the fault and performance data of each working node and the fault and performance data of the entire cluster, thereby realizing multi-level fault analysis.

[0050] Furthermore, the system also includes:

[0051] The display panel 15 is used to receive fault analysis reports from the analysis report unit 13 and display the fault analysis reports in a visual manner.

[0052] By visualizing the fault analysis report, technicians can gain a more intuitive and quick understanding of the cluster's working status.

[0053] Secondly, such as Figure 2 As shown, this embodiment of the invention provides a fault monitoring method for a K8S cluster, including:

[0054] Step S100: Select a target BCC template from the preset candidate BCC templates in the K8S cluster according to the target monitoring requirements, generate BCC tool CRD resources and BCC rule CRD resources according to the target BCC template, and submit the BCC tool CRD resources and BCC rule CRD resources to the K8S cluster.

[0055] Step S101: Pull the BCC tool CRD resource from the K8S cluster, parse the BCC tool CRD resource to obtain the BCC tool, run the BCC tool to monitor the working node, and obtain the monitoring result data on the working node.

[0056] Step S102: Pull the BCC rule CRD resource from the K8S cluster, parse the monitoring rules based on the BCC rule CRD resource, generate the Metrics of the working node according to the monitoring rules and the monitoring result data of the working node, and mark the Metrics as normal or abnormal according to the monitoring rules.

[0057] Step S103: Receive each metric from each working node, analyze all received metrics, and obtain a fault analysis report.

[0058] Furthermore, the method also includes:

[0059] Based on the received information on BCC tools and monitoring rules corresponding to specific monitoring needs, create at least one candidate BCC template and submit it to the K8S cluster.

[0060] Further, the BCC tool CRD resource is pulled from the K8S cluster, the BCC tool is obtained by parsing the BCC tool CRD resource, and the BCC tool is run to monitor the working node to obtain monitoring result data on the working node, including:

[0061] Pull the BCC tool CRD resource from the K8S cluster, and based on the BCC tool CRD resource, parse out the BCC tool corresponding to the target monitoring requirement on the working node.

[0062] On the working node, the BCC tool is run as a daemon to monitor the working node and collect monitoring results data.

[0063] Furthermore, the step of pulling the BCC rule CRD resource from the K8S cluster includes:

[0064] The BCC rule CRD resources are pulled from the K8S cluster by the Kubelet client program deployed on the working node;

[0065] The step of establishing normal or abnormal markers for the metrics according to monitoring rules includes:

[0066] Collect Metrics data on the working node and determine whether the monitoring result data encapsulated in the Metrics data has triggered the threshold defined by the monitoring rule, and whether the change value of the monitoring result data obtained in the current monitoring period relative to the monitoring result data of the previous fixed period exceeds the preset threshold. If it is determined that the monitoring result data encapsulated in the Metrics has triggered the threshold defined by the monitoring rule or the change value exceeds the preset threshold, then the Metrics data is marked as abnormal; otherwise, the Metrics data is marked as normal.

[0067] Furthermore, the method also includes:

[0068] The fault analysis report is presented in a visual manner.

[0069] The embodiments of the present invention are method embodiments that correspond one-to-one with the foregoing system embodiments. The embodiments of the present invention can be understood based on the foregoing system embodiments, and will not be repeated here.

[0070] The embodiments of this invention have the following technical effects: By defining BCC resource types, including BCC rule CRD resources and BCC rule CRD resources, through the CRD function in the Kubernetes system, and deploying monitoring devices and monitoring result collection devices through the DaemonSet method in the Kubernetes system, each worker node automatically runs the monitoring device and monitoring result collection device. Furthermore, each worker node can acquire BCC rule CRD resources and BCC rule CRD resources in the same way as acquiring standard Kubernetes resources. Combined with selecting the required target BCC template from the candidate BCC templates, multi-level monitoring is performed on the operating system, application software, container environment, and / or network environment, realizing multi-level anomaly analysis and detection for the Kubernetes system. By obtaining monitoring rule monitoring result data from the BCC rule CRD resources, automatic analysis and display of monitoring result data are achieved, avoiding the need for ordinary development, testing, and operation and maintenance technicians to directly analyze lengthy data, thus reducing the difficulty of use. Ordinary technicians do not need to be familiar with BCC tools, nor do they need to know how to run BCC tools on various working nodes in the K8S system. Ordinary technicians only need to select the appropriate target BCC template according to their own needs to be proficient in applying the fault detection function of the corresponding BCC tool in the K8S system, which lowers the technical threshold for ordinary technicians.

[0071] The technical solutions of the present invention will be described in detail below with reference to specific application examples. For technical details not described in the implementation process, please refer to the relevant descriptions above.

[0072] This system comprises multiple components, each responsible for a portion of the functionality. The system encapsulates the BCC tool and provides selectable BCC templates for deployment to the K8S cluster. It is started on each node of the cluster, making it more convenient to distribute the BCC tool to all compute nodes.

[0073] The components and relationships within this system are described below:

[0074] 1. K8S control terminal

[0075] like Figure 1 As shown, the Kubernetes cluster administrator is responsible for creating templates for the BCC tool, i.e., candidate BCC templates, for use by ordinary users. During the creation of candidate BCC templates, many details of the BCC tool are encapsulated, and parameters frequently used in historical experience are reserved for ordinary users to configure. The created candidate BCC templates are then submitted to the Kubernetes cluster, specifically to the Etcd key-value store system.

[0076] 2. K8S client

[0077] The application is deployed on a Kubernetes (K8S) cluster. To observe the application's running status on K8S or when an application encounters an anomaly, a target BCC template is obtained by selecting a candidate BCC template and configuring BCC parameters. Based on the target BCC template, BCC tool CRD resources and BCC rule CRD resources are generated and submitted to K8S. The BCC tool CRD resource is a K8S resource extension implemented in this system, containing attributes such as the BCC tool's name, owner, how it runs, how it returns data, and which nodes it runs on. The BCC rule CRD resource is also a K8S resource extension implemented in this system, containing restrictions on the BCC tool's return results, such as which CPU data and memory data are collected, which application layer protocols are parsed in network packets, which data is collected by the operating system, which BCC programs are disabled, and metric thresholds.

[0078] 3. BCC Client

[0079] The BCC client pulls the BCC tool CRD resources belonging to its current worker node from the Kubernetes cluster. The BCC client program is deployed across all nodes in the cluster using Kubernetes daemon sets. It then parses the BCC tool CRD resources into BCC tools. The BCC client program improves upon the original BCC usage method, which required starting a separate Python program from the BCC toolset via the command line. The original method did not collect the program's output.

[0080] The BCC client program improves upon the original startup method, handing over the task to the BCC daemon (equivalent to the BCC tool execution device) to take over all BCC tools and their program output, thereby automatically acquiring the monitoring results data output by the BCC tools.

[0081] 4. BCC daemon (i.e. Figure 1 BCC tool execution device in

[0082] The BCC client pushes received BCC tools to the BCC daemon, which is responsible for running and managing these tools. The daemon then retrieves monitoring data from the running BCC tools and pushes this data back to the BCC client.

[0083] 5. Kubelet monitoring extension (i.e. Figure 1 Metrics data generation device 122)

[0084] This component is an extension of kubelet. It receives monitoring results data from BCC tools pushed by the BCC client and filters and analyzes the data based on monitoring rules and results. It formats the data, tags it to identify its source, and encapsulates the monitoring rules and results into metrics data.

[0085] 6. kubelet (i.e. Figure 1 (Monitoring rule acquisition device in the middle)

[0086] This component is a Kubernetes client program deployed on each node. It is responsible for the current node's running status and executing Kubernetes commands, and this component supports custom extensions.

[0087] The kubelet pulls BCC rule CRD resources from Kubernetes and parses out the corresponding monitoring rules. It then sends the parsed monitoring rules to the kubelet monitoring extension component.

[0088] 7. Metrics Data Collector

[0089] The Metrics data collector receives metrics, checks the format of the metrics data, and removes abnormal or non-compliant metrics data. It then marks the thresholds in the monitoring rules onto the metrics to determine if the metrics have triggered the thresholds. Simultaneously, it compares the thresholds with values ​​from a fixed period ago; data exceeding the threshold is marked in red to indicate anomalies, while data within the threshold is marked in green to indicate normal operation.

[0090] 8. Analysis Reporter

[0091] The analysis reportr receives metric data from all nodes. Based on the abnormal metric data, it analyzes the operating system-level metrics, Kubernetes cluster software metrics, application metrics, and network metrics, and assigns priority to these metrics. Metrics with higher priority are more likely to be abnormal.

[0092] The analysis reportr sends out the analysis results, which is a report on the cause of the anomaly. The Kubernetes Metrics API automatically collects the kubelet's metrics and data analysis results.

[0093] The dashboard displays the relationships between different levels of metrics, connected by green lines. When data is abnormal, the green lines turn red, indicating an anomaly between the metrics and revealing the root cause of the anomaly.

[0094] The embodiments of this invention have the following technical effects: By defining BCC resource types through the CRD function in the Kubernetes system and deploying BCC clients via DaemonSet in the Kubernetes system, each designated node automatically runs a BCC client. Combined with the BCC program, monitoring targets including operating systems, application software, container environments, and / or network environments is achieved, enabling multi-level anomaly analysis and detection for the Kubernetes system. By analyzing the BCC program's results data through BCC rules, automatic analysis and display of detection results are achieved, avoiding the need for ordinary development, testing, and operations personnel to directly analyze lengthy data, thus reducing the difficulty of use. After deploying the application, each node can have BCC monitoring functionality by selecting a BCC template on the Kubernetes client, without needing to understand the details of the BCC program, lowering the barrier to entry. Kubernetes system deployment personnel can pre-create corresponding BCC templates according to various monitoring needs; other technical personnel do not need to be familiar with BCC tools, but can directly select the appropriate BCC template according to their own needs to proficiently use the fault detection function of the Kubernetes system.

[0095] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0096] In the above detailed description, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features of the single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, wherein each claim stands alone as a preferred embodiment of the invention.

[0097] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0098] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations falling within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is used in a manner similar to the term "including." Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

[0099] Those skilled in the art will also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, units, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0100] The various illustrative logic blocks or units described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0101] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0102] In one or more exemplary designs, the functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of these three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium in the form of one or more instructions or code. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a general-purpose or special-purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other forms that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Furthermore, any connection can be suitably defined as a computer-readable medium, for example, if the software is transmitted from a website, server or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disks and discs mentioned include compressed disks, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs. Disks typically copy data magnetically, while disks typically copy data optically using lasers. Combinations of the above can also be contained in computer-readable media.

[0103] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A fault monitoring system for a Kubernetes cluster, characterized in that, include: The K8S client is used to select a target BCC template from the preset candidate BCC templates in the K8S cluster according to the target monitoring requirements, generate BCC tool CRD resources and BCC rule CRD resources according to the target BCC template, and submit the BCC tool CRD resources and BCC rule CRD resources to the K8S cluster. The monitoring device is deployed on each working node in the K8S cluster. It is used to pull the BCC tool CRD resource from the K8S cluster, parse the BCC tool CRD resource to obtain the BCC tool, run the BCC tool to monitor the working node, obtain monitoring result data, and transmit the monitoring result data to the monitoring result collection device on the working node. The monitoring result collection device is deployed on each working node in the K8S cluster. It is used to pull the BCC rule CRD resource from the K8S cluster, parse the monitoring rule based on the BCC rule CRD resource, generate the metric of the working node according to the monitoring rule and the monitoring result data of the working node, establish normal or abnormal marking of the metric according to the monitoring rule, and send it to the analysis reporter. The analysis reportr is used to receive each metric from each working node, analyze all received metrics, and obtain a fault analysis report.

2. The fault monitoring system for a K8S cluster as described in claim 1, characterized in that, The system also includes: a K8S control terminal; The K8S control terminal is used to create at least one candidate BCC template based on the received information of BCC tools and monitoring rules corresponding to specific monitoring needs, and submit it to the K8S cluster.

3. The fault monitoring system for a K8S cluster as described in claim 1, characterized in that, The monitoring device includes: The BCC client is used to pull the BCC tool CRD resource from the K8S cluster, parse the BCC tool CRD resource to obtain the BCC tool corresponding to the target monitoring requirement, and submit the obtained BCC tool to the BCC tool execution device on the working node; and to receive monitoring result data from the BCC tool execution device on the working node and transmit the received monitoring result data to the monitoring result collection device on the working node. The BCC tool execution device is used to receive the BCC tool from the BCC client on the working node, run the BCC tool in the manner of a daemon process to monitor the working node, collect monitoring result data, and transmit the monitoring result data to the BCC client on the working node.

4. The fault monitoring system for a K8S cluster as described in claim 1, characterized in that, The monitoring result collection device includes: The monitoring rule acquisition device is used to pull the BCC rule CRD resource from the K8S cluster through the Kubelet client program deployed on the working node, parse the monitoring rules based on the BCC rule CRD resource, and transmit the parsed monitoring rules to the Metrics data generation device on the working node. Metrics data generation device is used to receive monitoring rules from the monitoring rule acquisition device on the working node and monitoring result data from the monitoring device on the working node, and encapsulate the received monitoring rules and monitoring result data into metric data; The Metrics data collector is used to collect Metrics data on the working node and determine whether the monitoring result data encapsulated in the Metrics data has triggered the threshold defined by the monitoring rule, and whether the change value of the monitoring result data obtained in the current monitoring period relative to the monitoring result data of the previous fixed period exceeds a preset threshold. If it is determined that the monitoring result data encapsulated in the Metrics has triggered the threshold defined by the monitoring rule or the change value exceeds the preset threshold, the Metrics data is marked as abnormal; otherwise, the Metrics data is marked as normal, and all Metrics are transmitted to the analysis reportr.

5. The fault monitoring system for a K8S cluster as described in claim 1, characterized in that, The system also includes: The display panel is used to receive fault analysis reports from the analysis reportr and to display the fault analysis reports in a visual manner.

6. A fault monitoring method for a Kubernetes cluster, characterized in that, include: According to the target monitoring requirements, a target BCC template is selected from the preset candidate BCC templates in the K8S cluster, and BCC tool CRD resources and BCC rule CRD resources are generated according to the target BCC template. The BCC tool CRD resources and BCC rule CRD resources are then submitted to the K8S cluster. Pull the BCC tool CRD resource from the K8S cluster, parse the BCC tool CRD resource to obtain the BCC tool, run the BCC tool to monitor the working node, and obtain the monitoring result data on the working node. Pull the BCC rule CRD resource from the K8S cluster, parse the monitoring rules based on the BCC rule CRD resource, generate the metric for the working node based on the monitoring rules and the monitoring result data of the working node, and mark the metric as normal or abnormal according to the monitoring rules. Receive each metric from each working node, analyze all received metrics, and obtain a fault analysis report.

7. The fault monitoring method for a K8S cluster as described in claim 6, characterized in that, The method further includes: Based on the received information on BCC tools and monitoring rules corresponding to specific monitoring needs, create at least one candidate BCC template and submit it to the K8S cluster.

8. The fault monitoring method for a K8S cluster as described in claim 6, characterized in that, The process involves pulling the BCC tool CRD resource from the Kubernetes cluster, parsing the BCC tool CRD resource to obtain the BCC tool, running the BCC tool to monitor the local worker node, and obtaining monitoring result data on the local worker node, including: Pull the BCC tool CRD resource from the K8S cluster, and based on the BCC tool CRD resource, parse out the BCC tool corresponding to the target monitoring requirement on the working node. On the working node, the BCC tool is run as a daemon to monitor the working node and collect monitoring results data.

9. The fault monitoring method for a K8S cluster as described in claim 6, characterized in that, The step of pulling the BCC rule CRD resource from the K8S cluster includes: The BCC rule CRD resources are pulled from the K8S cluster by the Kubelet client program deployed on the working node; The step of establishing normal or abnormal markers for the metrics according to monitoring rules includes: Collect Metrics data on the working node and determine whether the monitoring result data encapsulated in the Metrics data has triggered the threshold defined by the monitoring rule, and whether the change value of the monitoring result data obtained in the current monitoring period relative to the monitoring result data of the previous fixed period exceeds the preset threshold. If it is determined that the monitoring result data encapsulated in the Metrics has triggered the threshold defined by the monitoring rule or the change value exceeds the preset threshold, then the Metrics data is marked as abnormal; otherwise, the Metrics data is marked as normal.

10. The fault monitoring method for a K8S cluster as described in claim 6, characterized in that, The method further includes: The fault analysis report is presented in a visual manner.

Images