Processing method for fault discovery and recovery of intelligent calculation task

Abstract

An intelligent calculation task fault discovery and recovery processing method manages and dispatches an intelligent calculation task based on a Kubernetes cloud native platform, timely discovers an equipment fault through an equipment plug-in, simultaneously timely discovers the equipment fault in combination with an Exporters and Prometheus monitoring system, automatically shields a fault server and a fault calculation card through the dispatching plug-in, and automatically recovers the fault server and the fault calculation card through the dispatching plug-in. Meanwhile, according to the performance of the server and the performance of the computing card, the server deployed by the intelligent computing task and the bound computing card are preferably selected, and meanwhile, the equipment fault information and the performance information are stored on the Zookeeper, so that the intelligent computing task scheduling service can conveniently and timely carry out system change and timely rescheduling the task influenced by the fault; the intelligent calculation task influenced by the fault of the calculation card can be automatically recovered, the workload of operation and maintenance personnel can be reduced, full-automatic triggering is realized, human intervention is not needed when the calculation card has a fault, the fault calculation card is shielded, and the resource utilization rate is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent computing task fault processing, and in particular to a processing method for intelligent computing task fault discovery and recovery. BACKGROUND

[0002] With the rapid development of AI (artificial intelligence) and large models in recent years, the management and scheduling of intelligent computing tasks have become a hot technical problem. To improve processing efficiency, GPUs (graphics processing units), NPUs (neural network processors) and other computing cards are often used to accelerate the execution of intelligent computing tasks. As a result, the system becomes more complex and is more prone to faults, such as GPU card drop, parameter face network failure, network card failure, etc. Manual inspection is difficult to achieve comprehensive coverage and timely discovery of problems, and experience-based judgment may also deviate due to the complexity and specificity of intelligent computing systems. For example, traditional fault monitoring and recovery methods may not meet the real-time requirements, and once a fault occurs, the intelligent computing task will be interrupted and cannot be recovered for a long time.

[0003] Taking fault diagnosis as an example, the emergence of large model cluster computing tasks makes fault positioning in distributed systems inevitably face problems such as long positioning time and high complexity. Traditional operation and maintenance methods may take a lot of time to find the root cause of the problem, and if the business interruption time is too long, it will cause huge losses to the enterprise. In this way, fast handling of key business faults also puts higher requirements on the accuracy of system observation.

[0004] Existing technologies typically manage and schedule intelligent computing tasks through Kubernetes (a container cluster management system, container orchestration tool, or container orchestration engine), while monitoring the performance of hardware devices through technologies such as Prometheus (performance metric query interface), and then analyzing performance problems and generating device performance alarms; in addition, there are also ways to discover device fault information by monitoring system logs. After discovering device problems through these two methods, the operation and maintenance personnel will be notified to perform device maintenance. After the operation and maintenance personnel discover the fault, they need to manually restore the affected intelligent computing tasks. The specific steps are as follows: First, set the faulty server to an "unschedulable" state and isolate it to prevent subsequent new intelligent computing tasks from being scheduled to the faulty server; then, manually reschedule the intelligent computing tasks affected by the fault to other healthy servers. The disadvantages of existing technologies are as follows: (1) Low fault location efficiency. When a large-scale intelligent computing cluster experiences a performance decline, traditional monitoring methods may only display macro indicators such as "overall throughput decline", but cannot quickly locate which "slow card" or which "faulty link" caused the problem. (2) Manual handling of faults is required, and fault recovery is slow. The equipment failure was detected by monitoring, and then the intelligent computing task was restarted manually. This slowed down the recovery of the intelligent computing task affected by the failure, which is equivalent to an additional waste of computing resources. (3) Manually restoring the intelligent computing task requires marking the faulty server to prevent subsequent intelligent computing tasks from being rescheduled to the faulty server, resulting in low efficiency of intelligent computing task recovery. (4) Marking the faulty server may waste computing power. The failure may be a card on the server. Since there is no card-level fault isolation method, it can only be isolated by server, which wastes the computing resources of other cards on the same server. Summary of the Invention

[0005] This invention addresses the shortcomings or defects in existing technologies by providing a method for detecting and recovering intelligent computing task faults. This method facilitates the automatic recovery of intelligent computing tasks affected by computing card failures, reduces the workload of maintenance personnel, and is fully automatically triggered. It eliminates the need for human intervention when a computing card fails, shields the faulty computing card, and improves resource utilization.

[0006] The technical solution of the present invention is as follows:

[0007] A method for detecting and recovering from faults in intelligent computing tasks, characterized by comprising the following steps:

[0008] Step 1: In the Worker server of the intelligent computing system, the device plugin is used to detect intelligent computing task faults. After the intelligent computing task fault is detected, it is reported to the Kubelet component through the Event interface, and then transmitted to the system monitoring service module by the Kubelet component.

[0009] Step 2: The device plugin updates the device fault status on Zookeeper for the device that caused the intelligent computing task failure;

[0010] Step 3: Zookeeper reports the intelligent computing task failure to the intelligent computing task scheduling service module;

[0011] Step 4: The intelligent computing task scheduling service module reschedules the intelligent computing tasks affected by the failure through the Kubernetes Master scheduling interface of the scheduling plugin.

[0012] Step 5: After receiving the intelligent computing task, the Kubernetes Master first calls the scheduling plugin to perform node filtering, filtering out server nodes that meet the resource requirements of the intelligent computing task, and filtering out server nodes with server-level failures.

[0013] Step 6: Determine if there is a server that meets the conditions. If not, put the intelligent computing task into a pending state and wait for the next scheduling cycle to return to step 4 for rescheduling. If yes, proceed to step 7.

[0014] Step 7: The Kubernetes Master sorts the servers that meet the criteria, ranking them higher if the criteria are better.

[0015] Step 8: The Kubernetes Master distributes the intelligent computing task to the Kubelet component;

[0016] Step 9: The Kubelet component starts the intelligent computing task and calls the device plugin to bind the computing card to the server with the highest ranking to execute the intelligent computing task. The device plugin automatically blocks faulty computing cards and selects available computing cards to allocate to the intelligent computing task based on the device status.

[0017] Step 7 includes sorting the servers by scoring them. The server score is calculated as follows: 1 minute memory utilization rate + 10 minutes memory utilization rate * 1.5 + 30 minutes memory utilization rate * 2.5 + number of faulty cards * 100. The servers are then sorted in descending order of their scores.

[0018] In step 1, the system monitoring service module will notify the operation and maintenance personnel of alarms derived from performance indicators and faults directly caused by the equipment via SMS, reminding the operation and maintenance personnel to repair the faults.

[0019] In step 1, the device plugin is responsible for discovering the computing card, monitoring its operating status in real time, promptly identifying computing card faults, and registering the computing card resources to the Kubernetes cluster, which then manages the computing card resources on each server in a unified manner.

[0020] In step 1, the system monitoring service module collects fault events reported by Kubelet. If Kubelet is unreachable, it is determined that the server node is faulty.

[0021] In step 3, the intelligent computing task scheduling service module registers a Watch for device information on Zookeeper. When the device status changes, Zookeeper notifies the intelligent computing task scheduling service of the device status through the Watch mechanism.

[0022] In step 1, the system monitoring service module connects to the Exporters performance acquisition component in the Worker server through the Prometheus performance metric query interface. The Exporters collect the performance status of the server or chip and report the performance data to Prometheus. The system monitoring service module generates performance alarms based on the performance data from Prometheus.

[0023] The technical effects of this invention are as follows: This invention provides a method for detecting and recovering intelligent computing task faults. Based on the Kubernetes cloud-native platform for managing and scheduling intelligent computing tasks (Kubernetes, a container cluster management system, container orchestration tool, or container orchestration engine), it promptly detects device faults through device plugins and combines Exporters (performance acquisition components) and Prometheus (performance metric query interface) monitoring systems to promptly detect device faults. The scheduling plugin automatically masks faulty servers and computing cards, and optimizes the servers and bound computing cards for intelligent computing task deployment based on server and computing card performance. Simultaneously, device fault information and performance information are saved to Zookeeper (a distributed server), facilitating timely watch (event notification mechanism) of system changes by the intelligent computing task scheduling service and enabling timely rescheduling of tasks affected by faults.

[0024] Compared with the prior art, the present invention has the following characteristics:

[0025] (1) The mechanism and approach for automatic recovery of computing card failures through device plugins, intelligent computing task scheduling service, and scheduling plugins. The device plugins quickly detect failures, the intelligent computing task scheduling service automatically reschedules the intelligent computing tasks affected by the failures, and the scheduling plugins filter and select appropriate server nodes. The scheduling plugins automatically mark the faulty computing cards.

[0026] (2) A fast rescheduling algorithm is provided through ZooKeeper's watch mechanism. After a fault is detected, it is written to ZooKeeper, and ZooKeeper quickly notifies the intelligent computing task scheduling service through the watch mechanism. When scheduling tasks, the scheduling plugin provides a decision-making algorithm by reading device-related information in ZooKeeper. At the same time, ZooKeeper is a distributed component, which ensures the reliability and consistency of device information.

[0027] (3) The algorithm mechanism for scoring the server is based on the computing card load and computing card failure status. The percentage of video memory usage of the computing card is used as the basis for the computing card load status. At the same time, the failure card is regarded as having a video memory usage of 100. The formula is: 1 minute video memory utilization rate + 10 minutes video memory utilization rate * 1.5 + 30 minutes video memory utilization rate * 2.5 + number of failure cards * 100. Then, the scores are sorted in reverse order. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the intelligent computing system structure involved in the intelligent computing task fault detection and recovery processing method of the present invention. Figure 1 It includes a system monitoring service module, which connects to both the Worker server and the Zookeeper cluster (i.e., a distributed server cluster containing device information, Device Info). The Zookeeper cluster is connected to a scheduling plugin (which has Kubernetes functionality). The Master (container cluster management system cluster control node), the intelligent computing task scheduling service module, and the device plugins in the Worker server are all involved. These device plugins obtain status monitoring data from computing cards such as GPUs / NPUs / MLUs, report faults to the Kubelet component via the Event interface, transmit updated device status data to the Zookeeper cluster, and the Zookeeper cluster transmits queried device status data to the scheduling plugin. Performance data fault information is transmitted (using a Watch event notification mechanism) to the intelligent computing task scheduling service module, which receives device performance data updates from the system monitoring service module. The intelligent computing task scheduling service module transmits intelligent computing task rescheduling instructions to the scheduling plugin. The system monitoring service module receives Event collection information from the Kubelet component, which starts / stops / monitors intelligent computing tasks. The system monitoring service module connects to the Exporters performance collection component in the Worker server via the Prometheus performance metric query interface. The Exporters report performance data to the system monitoring service module via Prometheus. The system monitoring service module notifies maintenance personnel of faults via SMS and other means for timely maintenance.

[0029] Figure 2 This is a schematic diagram of the process for detecting and recovering faults in intelligent computing tasks according to the present invention. Figure 2 The process includes: Step 1, the device plugin detects a fault and reports it to the Kubelet component via the Event interface; Step 2, the device plugin updates the device fault status on Zookeeper; Step 3, Zookeeper reports the fault to the intelligent computing task scheduling service module; Step 4, the intelligent computing task scheduling service module reschedules the intelligent computing tasks affected by the fault through the Kubernetes Master scheduling interface of the scheduling plugin; Step 5, after receiving the intelligent computing task, the Kubernetes Master first calls the scheduling plugin to perform node filtering, filtering out server nodes that meet the resource requirements of the intelligent computing task, while filtering out server nodes with server-level faults; Step 6, it determines whether there is a server that meets the conditions. If not, the intelligent computing task is put into a pending state, waiting for the next scheduling cycle to return to Step 4 for rescheduling. If yes, it proceeds to Step 7; Step 7, the Kubernetes Master performs server sorting; Step 8, the Kubernetes Master issues the intelligent computing task to the Kubelet component; Step 9, the Kubelet component starts the intelligent computing task. Detailed Implementation

[0030] The following is in conjunction with the attached diagram ( Figures 1-2 The present invention will be described in conjunction with the examples.

[0031] Figure 1 This is a schematic diagram of the intelligent computing system structure involved in the intelligent computing task fault detection and recovery processing method of the present invention. Figure 2 This is a schematic flowchart illustrating a method for detecting and recovering faults in intelligent computing tasks according to the present invention. (Reference) Figures 1-2As shown, a method for detecting and recovering intelligent computing task faults includes the following steps: Step 1, the device plugin in the Worker server of the intelligent computing system detects intelligent computing task faults. Once detected, the fault is reported to the Kubelet component via the Event interface, and then transmitted to the system monitoring service module by the Kubelet component; Step 2, the device plugin updates the device fault status on Zookeeper for the device causing the intelligent computing task fault; Step 3, Zookeeper reports the intelligent computing task fault to the intelligent computing task scheduling service module; Step 4, the intelligent computing task scheduling service module reschedules the affected intelligent computing tasks through the Kubernetes Master scheduling interface of the scheduling plugin; Step 5, after receiving the intelligent computing task, the Kubernetes Master first calls the scheduling plugin to perform node filtering, filtering out server nodes that meet the resource requirements of the intelligent computing task, while filtering out server nodes with server-level faults; Step 6, it determines whether there is a server that meets the conditions. If not, the intelligent computing task is put into a pending state, waiting for the next scheduling cycle to return to Step 4 for rescheduling; if yes, it proceeds to Step 7; Step 7, Kubernetes... The Master sorts the servers that meet the conditions, with better conditions resulting in higher rankings. Step 8: The Kubernetes Master distributes intelligent computing tasks to the Kubelet component. Step 9: The Kubelet component starts the intelligent computing task and calls the device plugin to bind the computing card to the server with the highest ranking to execute the intelligent computing task. The device plugin automatically blocks faulty computing cards based on the device status and selects available computing cards to allocate to the intelligent computing task.

[0032] Step 7 includes sorting the servers by scoring them. The server score is calculated as follows: 1 minute memory utilization rate + 10 minutes memory utilization rate * 1.5 + 30 minutes memory utilization rate * 2.5 + number of faulty cards * 100. The servers are then sorted in descending order of their scores.

[0033] In Step 1, the system monitoring service module notifies operations and maintenance personnel via SMS of performance-related alarms and device-related faults, reminding them to repair the faults. The device plugin in Step 1 is responsible for discovering compute cards, monitoring their operational status in real time, promptly detecting compute card faults, and registering the compute card resources to the Kubernetes cluster for unified management of compute card resources on various servers. The system monitoring service module in Step 1 also collects fault events reported by the Kubelet; if the Kubelet is unreachable, it indicates a server node failure.

[0034] In step 3, the intelligent computing task scheduling service module registers a Watch for device information on ZooKeeper. When the device status changes, ZooKeeper notifies the intelligent computing task scheduling service of the device status through the Watch mechanism. In step 1, the system monitoring service module connects to the Exporters performance acquisition component in the Worker server through the Prometheus performance metric query interface. The Exporters collect the performance status of the server or chip and report the performance data to Prometheus. The system monitoring service module generates performance alarms based on the Prometheus performance data.

[0035] refer to Figure 1 The system is built using Kubernetes (Kubernetes is a container cluster management system, container orchestration tool, or container orchestration engine) to manage servers that execute intelligent computing tasks. These servers have computing cards such as GPUs, NPUs, and MLUs (GPU, Graphics Processing Unit; NPU, Neural Network Processing Unit; MLU, Machine Learning Unit) to provide AI computing power (AI, Artificial Intelligence), and are usually called Worker servers. The Kubernetes Master is responsible for managing several Worker servers (Master, cluster control node) and scheduling intelligent computing tasks to different Worker servers according to certain strategies.

[0036] Exporters are performance acquisition components for various devices, responsible for collecting performance metrics related to the execution of servers, computing cards, CPUs (Central Processing Units), memory, and intelligent computing tasks. Exporters then output the collected performance metrics to Prometheus, which is responsible for storing various performance metrics within the system and providing a performance metric query interface.

[0037] The device plugin is responsible for discovering compute cards and registering their resources with the Kubernetes cluster, which then manages the compute card resources on each server. In addition to registering compute card resources, this patent also requires the device plugin to periodically check the status of the compute cards, write their resource and fault information to the Zookeeper cluster (Zookeeper, a distributed server), and report fault events through the Event interface of the Kubelet component in the Kubernetes cluster.

[0038] The system monitoring service simultaneously collects Prometheus performance metrics and Kubelet event information. It also writes the performance metric information to the Zookeeper cluster. Furthermore, the service notifies operations and maintenance personnel via SMS and other means of receiving performance metric-related alarms and directly caused device faults, reminding them to resolve the issues.

[0039] The intelligent computing task scheduling service monitors server and computing card failure information through Zookeeper's Watch function (i.e., event notification mechanism). When a failure is detected, the intelligent computing tasks on the failed server or those bound to the failed computing card are automatically rescheduled. The intelligent computing task scheduling service sends the rescheduling command to the Kubernetes Master node.

[0040] The scheduling plugin implements two main categories of functions: filtering and sorting. When scheduling tasks, the Kubernetes Master node first uses the filtering function of the scheduling plugin to select servers that meet the resource requirements of the intelligent computing tasks, while filtering out faulty server nodes. Then, the sorting function of the scheduling plugin sorts the server nodes, placing servers with faulty cards but not a complete server failure at a lower priority, and using fault-free server nodes as much as possible. If resources are scarce, intelligent computing tasks can be scheduled to non-faulty computing cards on faulty servers.

[0041] The normal data collection and reporting process is as follows.

[0042] 1. Various exporters collect server or chip performance status and report the performance data to Prometheus.

[0043] 2. The system monitoring service generates performance alerts based on Prometheus performance data.

[0044] 3. The device plugin monitors the computing card's operating status in real time, promptly detecting computing card faults and parameter network failures. Upon detecting a fault, the device plugin immediately reports it to the Kubelet on the local server node and updates the corresponding device status on ZooKeeper.

[0045] 4. The system monitoring service collects fault events reported by Kubelet. If Kubelet is unreachable, it is determined that the server node is faulty.

[0046] 5. The system monitoring service will send fault information to maintenance personnel via SMS and other means, notifying them to handle equipment faults as soon as possible.

[0047] 6. The intelligent computing task scheduling service has registered a Watch (event notification mechanism) for device information on Zookeeper. When the device status changes, Zookeeper will notify the intelligent computing task scheduling service of the device status.

[0048] refer to Figure 2 When a fault occurs, the business processing flow is as follows:

[0049] 1. When a device plugin detects a fault, it reports it to the Kubelet via the Event interface.

[0050] 2. Update the device's fault status on Zookeeper using the device plugin.

[0051] 3. Zookeeper reports device fault status to the intelligent computing task scheduling service through the Watch interface.

[0052] 4. The intelligent computing task scheduling service reschedules intelligent computing tasks affected by computing card failures. The rescheduling is achieved by calling the scheduling interface of the Kubernetes Master.

[0053] 5. After receiving the intelligent computing task scheduling call, the Kubernetes Master first calls the scheduling plugin to perform node filtering, filtering out server nodes that meet the resource requirements of the intelligent computing task, and filtering out server nodes with server-level failures. If no server meets the requirements of the intelligent computing task after filtering, the current intelligent computing task enters a pending state (i.e., suspended or waiting state), waiting for the next scheduling cycle to be rescheduled.

[0054] 6. The Kubernetes Master then calls the scheduling plugin to sort the filtered servers. The sorting is done using a scoring method, and the servers are sorted in descending order of their scores. The server scoring formula is: 1-minute memory utilization + 10-minute memory utilization * 1.5 + 30-minute memory utilization * 2.5 + number of faulty cards * 100.

[0055] 7. The Kubernetes Master selects the server with the highest ranking to execute the intelligent computing task and distributes the intelligent computing task to the Kubelet. It also specifies the computing card ID (identification number) to which the intelligent computing task is bound.

[0056] 8. Kubelet starts the intelligent computing task and calls the device plugin to bind the computing card to the intelligent computing task. The device plugin automatically disables faulty computing cards based on the device status, selects available computing cards to assign to the intelligent computing task, and binds the computing card to the intelligent computing task container.

[0057] Compared with the prior art, the technical advantages of the present invention are as follows:

[0058] 1. This invention can automatically recover intelligent computing tasks affected by computing card failure.

[0059] In existing technical solutions, after a fault is detected by a monitoring system, personnel manually restore the affected intelligent computing tasks. Adopting an automatic intelligent computing task recovery mechanism can quickly restore faulty intelligent computing tasks, avoiding resource waste and improving customer satisfaction.

[0060] 2. This invention can reduce the workload of maintenance personnel, is fully automatic, and does not require human intervention when the computing card fails.

[0061] In existing technical solutions, after a failure occurs, the faulty server needs to be manually marked to prevent subsequent intelligent computing tasks from being incorrectly scheduled to the faulty server. After the failure is resolved, the faulty server also needs to be manually unmarked.

[0062] 3. This invention can use a dynamic algorithm to score the server and automatically block faulty calculation cards, thereby improving resource utilization.

[0063] Existing technical solutions rely on manual fault recovery, which only marks the faulty server, rendering other computing cards on the same server unusable and wasting computing power. The current method can intelligently score servers and automatically isolate faulty computing cards during scheduling, achieving full utilization of computing power.

[0064] Contents not described in detail in this specification are prior art known to those skilled in the art. It is hereby indicated that the above description is intended to help those skilled in the art understand this invention, but does not limit the scope of protection of this invention. Any equivalent substitutions, modifications, improvements, and / or simplifications of the above descriptions that do not depart from the essential content of this invention fall within the scope of protection of this invention.

Claims

1. A method for detecting and recovering from faults in intelligent computing tasks, characterized in that, Includes the following steps: Step 1: In the Worker server of the intelligent computing system, the device plugin is used to detect intelligent computing task faults. After the intelligent computing task fault is detected, it is reported to the Kubelet component through the Event interface, and then transmitted to the system monitoring service module by the Kubelet component. Step 2: The device plugin updates the device fault status on Zookeeper for the device that caused the intelligent computing task failure; Step 3: Zookeeper reports the intelligent computing task failure to the intelligent computing task scheduling service module; Step 4: The intelligent computing task scheduling service module reschedules the intelligent computing tasks affected by the failure through the Kubernetes Master scheduling interface of the scheduling plugin. Step 5: After receiving the intelligent computing task, the Kubernetes Master first calls the scheduling plugin to perform node filtering, filtering out server nodes that meet the resource requirements of the intelligent computing task, and filtering out server nodes with server-level failures. Step 6: Determine if there is a server that meets the conditions. If not, put the intelligent computing task into a pending state and wait for the next scheduling cycle to return to step 4 for rescheduling. If yes, proceed to step 7. Step 7: The Kubernetes Master sorts the servers that meet the criteria, ranking them higher if the criteria are better. Step 8: The Kubernetes Master distributes the intelligent computing task to the Kubelet component; Step 9: The Kubelet component starts the intelligent computing task and calls the device plugin to bind the computing card to the server with the highest ranking to execute the intelligent computing task. The device plugin automatically blocks faulty computing cards and selects available computing cards to allocate to the intelligent computing task based on the device status.

2. The method for detecting and recovering faults in intelligent computing tasks according to claim 1, characterized in that, Step 7 includes sorting the servers by scoring them. The server score is calculated as follows: 1 minute memory utilization rate + 10 minutes memory utilization rate * 1.5 + 30 minutes memory utilization rate * 2.5 + number of faulty cards * 100. The servers are then sorted in descending order of their scores.

3. The method for detecting and recovering faults in intelligent computing tasks according to claim 1, characterized in that, In step 1, the system monitoring service module will notify the operation and maintenance personnel of alarms derived from performance indicators and faults directly caused by the equipment via SMS, reminding the operation and maintenance personnel to repair the faults.

4. The method for detecting and recovering faults in intelligent computing tasks according to claim 1, characterized in that, In step 1, the device plugin is responsible for discovering the computing card, monitoring its operating status in real time, promptly identifying computing card faults, and registering the computing card resources to the Kubernetes cluster, which then manages the computing card resources on each server in a unified manner.

5. The method for detecting and recovering faults in intelligent computing tasks according to claim 1, characterized in that, In step 1, the system monitoring service module collects fault events reported by Kubelet. If Kubelet is unreachable, it is determined that the server node is faulty.

6. The method for detecting and recovering faults in intelligent computing tasks according to claim 1, characterized in that, In step 3, the intelligent computing task scheduling service module registers a Watch for device information on Zookeeper. When the device status changes, Zookeeper notifies the intelligent computing task scheduling service of the device status through the Watch mechanism.

7. The method for detecting and recovering faults in intelligent computing tasks according to claim 1, characterized in that, In step 1, the system monitoring service module connects to the Exporters performance acquisition component in the Worker server through the Prometheus performance metric query interface. The Exporters collect the performance status of the server or chip and report the performance data to Prometheus. The system monitoring service module generates performance alarms based on the performance data from Prometheus.

Images