Heterogeneous GPU server cluster monitoring system and method based on Zabbix and Grafana

Abstract

The invention relates to the technical field of information technology and operation and maintenance monitoring, in particular to a heterogeneous GPU (Graphics Processing Unit) server cluster monitoring system and method based on Zabbix and Grafana, a layered architecture design is adopted, the heterogeneous GPU server cluster monitoring system comprises four hierarchies, data interaction is performed among the hierarchies through a network, and a complete, automatic and visual distributed monitoring system is jointly formed; the method has the beneficial effects that heterogeneous GPUs are supported, multi-model GPUs are monitored in a unified manner, and the compatibility and the applicability are improved; the automation degree is high, automatic collection, reporting and alarming are achieved, and manual intervention is reduced; the expansibility is high, and monitoring expansion of hundreds of GPU servers is supported; the visualization effect is good; multi-dimensional data display and historical trend analysis are carried out; the operation and maintenance efficiency is improved; one-key report, automatic alarm and resource optimization support are realized.

Description

Technical Field

[0001] This invention relates to the fields of information technology and operation and maintenance monitoring technology, specifically to a heterogeneous GPU server cluster monitoring system and method based on Zabbix and Grafana. Background Technology

[0002] With the development of artificial intelligence and high-performance computing, GPU server clusters are expanding in scale and GPU models are becoming increasingly diverse, making it difficult for traditional monitoring solutions to achieve unified management of heterogeneous GPU resources. Existing technologies suffer from complex configurations, poor scalability, and weak visualization capabilities when monitoring multiple machines and multiple GPU models. While Zabbix has strong data acquisition and alerting capabilities, it lacks native support for heterogeneous GPUs. Grafana can serve as a visualization supplement, but currently, a complete heterogeneous GPU cluster monitoring solution is lacking.

[0003] Especially when dealing with monitoring scenarios involving more than 100 heterogeneous GPU servers, existing technologies face the following challenges: (1) inability to automatically discover and register multiple GPU nodes; (2) lack of a unified tool for collecting heterogeneous GPU metrics; (3) low level of visualization of monitoring data; (4) imperfect alarm mechanism; and (5) weak historical data analysis capabilities. Therefore, there is an urgent need for a unified monitoring solution that can support multiple GPU models. Summary of the Invention

[0004] The purpose of this invention is to provide a heterogeneous GPU server cluster monitoring system and method based on Zabbix and Grafana, which realizes automatic collection, storage, alarm and visualization of key indicators such as video memory and GPU utilization of multiple GPU servers, improves the level of operation and maintenance automation and system observability of heterogeneous GPU clusters, and solves the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a heterogeneous GPU server cluster monitoring system based on Zabbix and Grafana, the system adopting a layered architecture design, including: The data acquisition layer consists of the Zabbix Agent deployed on the GPU server and acquisition scripts for various GPU models. Data storage and analysis layer: Composed of Zabbix Server and database, responsible for data processing and alarm analysis; Data visualization and alerting layer: Includes Grafana and Zabbix Web UI, responsible for data visualization and alert notification; User and Operations Layer: Provides monitoring insights and decision support for operations personnel; Heterogeneous GPU device layer: including various models of GPU devices from NVIDIA and Tianshu Zhixin.

[0006] Preferably, the data acquisition layer supports monitoring data acquisition from various GPU chips, including NVIDIA GPUs and Tianshu Zhixin GPUs, and achieves adaptation through a unified script interface.

[0007] Preferably, the data storage and analysis layer supports horizontal scaling by adding Zabbix Proxy nodes to share the data collection burden.

[0008] Preferably, the system supports automated report generation and distribution, and integrates with other operation and maintenance systems through a REST API interface.

[0009] A method for monitoring heterogeneous GPU server clusters based on Zabbix and Grafana includes the following steps: Deploy Zabbix Server and Grafana on the monitoring server, and configure the database and network ports; Deploy the Zabbix Agent on each GPU server and collect GPU metrics using custom scripts adapted to multiple GPU models; Configure Zabbix Agent user parameters to enable Zabbix Server to pull GPU monitoring data; In Zabbix, create a monitoring template and host group, and configure monitoring items and triggers; Configure a Zabbix data source in Grafana to create a visualization dashboard; Configure alarm rules and report export functions to achieve automated operation and maintenance.

[0010] Preferably, the custom script supports multiple GPU models, including NVIDIA GPUs and Tianshu Zhixin GPUs, and automatically calls the corresponding commands to collect data.

[0011] Preferably, Grafana generates PDF reports through the Grafana Reporter plugin, which supports exporting monitoring data by time range.

[0012] Preferably, the Zabbix Agent configuration includes automatic node discovery, automatic registration, and dynamic hostname configuration, supporting automated monitoring and access to large-scale heterogeneous GPU clusters.

[0013] Preferably, the monitoring metrics include multi-dimensional data such as GPU utilization, video memory usage, temperature, and power consumption, which are then uniformly displayed and alerted through Grafana.

[0014] Compared with the prior art, the beneficial effects of the present invention are: The heterogeneous GPU server cluster monitoring system and method proposed in this invention based on Zabbix and Grafana supports heterogeneous GPUs: unified monitoring of multiple GPU models, improving compatibility and applicability; high degree of automation: automatic data collection, reporting, and alarming, reducing manual intervention; strong scalability: supporting monitoring expansion to hundreds of GPU servers; good visualization effect: multi-dimensional data display and historical trend analysis; improved operation and maintenance efficiency: one-click reporting, automatic alarms, and resource optimization support. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the present invention clear and complete, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some, not all, embodiments of the present invention, and are merely illustrative of the embodiments of the present invention. They are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1: This invention provides a technical solution: a heterogeneous GPU server cluster monitoring system based on Zabbix and Grafana. The system adopts a layered architecture design, including the following four layers. Each layer interacts with the others via a network, collectively forming a complete, automated, and visualized distributed monitoring system: 1. Data Acquisition Layer (Agent Layer): This layer consists of Zabbix Agents deployed on each heterogeneous GPU server (such as servers equipped with different models of GPUs, such as NVIDIA and Tianshu Zhixin), and forms the basis for monitoring data sources.

[0018] The Agent automatically identifies the GPU model and calls the corresponding hardware commands (such as nvidia-smi or ixsmi) by executing a unified custom script (such as gpu_stats.sh) to collect key indicators such as GPU utilization, memory usage, and memory efficiency.

[0019] The collected data is reported to the data storage and analysis layer via the Zabbix Agent in an active or passive manner, realizing a unified collection interface for GPU metrics from multiple manufacturers.

[0020] 2. Data Storage and Analysis Layer (Server Layer): This layer, centered around the Zabbix Server, is responsible for receiving, processing, storing, and analyzing the massive amounts of heterogeneous GPU monitoring data reported from the Agent layer.

[0021] The MySQL / MariaDB database serves as the backend storage, persistently storing all historical monitoring data and configuration information (such as host lists, monitoring items, triggers, etc.).

[0022] Zabbix Server analyzes data in real time based on pre-configured trigger rules, and generates an alarm event once an anomaly is detected (such as the utilization of a certain type of GPU exceeding a threshold).

[0023] 3. Data Presentation and Alert Layer: This layer contains two core components: Grafana and Zabbix Web UI, which are responsible for data visualization and system configuration management.

[0024] Grafana: By configuring a Zabbix data source, it retrieves data from the Zabbix Server API and provides powerful dashboard functionality, displaying multi-dimensional data and analyzing historical trends of heterogeneous GPU clusters in rich chart formats (such as line charts, dashboards, tables, etc.).

[0025] Zabbix Web UI: Primarily used for the daily configuration and management of monitoring systems, such as managing hosts and templates, configuring monitoring items and alarm rules, and viewing alarm events.

[0026] Alarm information can be sent to operations and maintenance personnel through various means such as email and webhooks, enabling timely notification.

[0027] 4. User Layer: Operations engineers and business managers can access Grafana dashboards and Zabbix Web UIs through a browser to monitor the overall health status of heterogeneous GPU clusters, the performance of each GPU model, and resource utilization in real time, thereby making efficient operations decisions and resource planning.

[0028] Example 2, based on Example 1, proposes a method for monitoring heterogeneous GPU server clusters using Zabbix and Grafana, including the following steps: Deploy Zabbix Server and Grafana on the monitoring server, and configure the basic environment such as database, network port, and Chinese language support; Install the Zabbix Agent on each GPU server and collect GPU metrics using custom scripts that are compatible with multiple vendors (such as gpu_stats.sh). Configure Zabbix Agent user parameters to enable Zabbix Server to actively pull GPU monitoring data; Create monitoring templates and host groups in Zabbix to manage heterogeneous GPU servers in a unified manner, and configure monitoring items, triggers, graphs and dashboards; By connecting Grafana to Zabbix data sources, you can import or customize dashboards to achieve multi-dimensional visualization. Configure alarm rules and report export functions, and support generating PDF reports via Grafana Reporter.

[0029] Achieve unified monitoring of multiple heterogeneous GPU models: Automatically identify and adapt to GPU devices from different manufacturers such as NVIDIA (nvidia-smi) and ixsmi through scripts; Dynamic configuration and automatic discovery are achieved: Zabbix Agent automatically obtains the local business IP and completes the configuration, supporting automatic registration for large-scale clusters; Implement a layered monitoring architecture: A four-layer architecture is adopted to separate concerns and reduce system coupling; Achieve unified report output: Integrates Grafana-reporter, supporting one-click export of PDF reports.

[0030] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A heterogeneous GPU server cluster monitoring system based on Zabbix and Grafana, characterized in that: The system adopts a layered architecture design, including: The data acquisition layer consists of the Zabbix Agent deployed on the GPU server and acquisition scripts for various GPU models. Data storage and analysis layer: Composed of Zabbix Server and database, responsible for data processing and alarm analysis; Data visualization and alerting layer: Includes Grafana and Zabbix Web UI, responsible for data visualization and alert notification; User and Operations Layer: Provides monitoring insights and decision support for operations personnel; Heterogeneous GPU device layer: including various models of GPU devices from NVIDIA and Tianshu Zhixin.

2. The heterogeneous GPU server cluster monitoring system based on Zabbix and Grafana according to claim 1, characterized in that: The data acquisition layer supports monitoring data acquisition from various GPU chips, including NVIDIA GPUs and Tianshu Zhixin GPUs, and achieves adaptation through a unified script interface.

3. The heterogeneous GPU server cluster monitoring system based on Zabbix and Grafana according to claim 2, characterized in that: The data storage and analysis layer supports horizontal scaling by adding Zabbix Proxy nodes to share the data collection burden.

4. The heterogeneous GPU server cluster monitoring system based on Zabbix and Grafana according to claim 3, characterized in that: The system supports automated report generation and distribution, and integrates with other operation and maintenance systems through a REST API interface.

5. A method for monitoring heterogeneous GPU server clusters based on Zabbix and Grafana, using the system described in claim 4, characterized in that: The method includes the following steps: Deploy Zabbix Server and Grafana on the monitoring server, and configure the database and network ports; Deploy the Zabbix Agent on each GPU server and collect GPU metrics using custom scripts adapted to multiple GPU models; Configure Zabbix Agent user parameters to enable Zabbix Server to pull GPU monitoring data; In Zabbix, create a monitoring template and host group, and configure monitoring items and triggers; Configure a Zabbix data source in Grafana to create a visualization dashboard; Configure alarm rules and report export functions to achieve automated operation and maintenance.

6. The method for monitoring heterogeneous GPU server clusters based on Zabbix and Grafana according to claim 5, characterized in that: The custom script supports multiple GPU models, including NVIDIA GPUs and Tianshu Zhixin GPUs, and automatically calls the corresponding commands to collect data.

7. A method for monitoring heterogeneous GPU server clusters based on Zabbix and Grafana according to claim 6, characterized in that: Grafana generates PDF reports through the Grafana Reporter plugin and supports exporting monitoring data by time range.

8. The method for monitoring heterogeneous GPU server clusters based on Zabbix and Grafana according to claim 7, characterized in that: Zabbix Agent configuration includes automatic node discovery, automatic registration, and dynamic hostname configuration, supporting automated monitoring and access to large-scale heterogeneous GPU clusters.

9. A method for monitoring heterogeneous GPU server clusters based on Zabbix and Grafana according to claim 8, characterized in that: The monitoring metrics include GPU utilization, video memory usage, temperature, power consumption, and other multi-dimensional data, which are displayed and alerted in a unified manner through Grafana.

Images