Method for managing and controlling computing power unit management service and electronic device

Abstract

The application discloses a kind of computing unit management service management and control method and electronic equipment, it is related to computer technical field, including establishing the control node between the target cluster and the node to be installed between password-free communication, and make control node possess the administrator operation authority of node to be installed;Target operating environment is deployed on control node, so that control node has the ability to batch control to node to be installed;For any node to be installed, based on the environment parameters of the node to be installed, target computing unit management service installation package is selected from computing unit management service installation package resource pool;Target computing unit management service installation package is sent to the node to be installed;In the node to be installed, target computing unit management service installation package is decompressed, and target computing unit management service is started.The technical problem that computing unit management service deployment efficiency is low in the related art is solved, and the technical effect that the deployment efficiency of computing unit management service is improved is achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a control method and electronic device for computing power unit management services. Background Technology

[0002] With the rapid development of heterogeneous computing technology and the continuous expansion of heterogeneous computing power clusters, stable and efficient monitoring of the performance and status of computing power units has become crucial, as computing power units are the core computing power support for heterogeneous computing. Computing power unit management services, as a lightweight computing power unit monitoring tool developed using the Go language, are widely used in single-node computing power unit monitoring due to their low resource consumption, read-only operation that does not affect system operation, and support for critical error detection. However, in heterogeneous computing power cluster scenarios, the deployment of computing power unit management services relies on manual installation and configuration on each node, which is cumbersome and prone to deployment failure due to differences in node environments, resulting in low deployment efficiency and high maintenance costs. Summary of the Invention

[0003] This application provides a control method and electronic device for computing unit management services, in order to at least solve the problems of low deployment efficiency and high operation and maintenance costs of computing unit management services in related technologies.

[0004] This application provides a management and control method for computing unit management services, including: Establish passwordless communication between the control node of the target cluster and the node to be installed in the target cluster, and grant the control node administrator operation privileges for the node to be installed. Deploy the target runtime environment on the control node, which enables the control node to perform batch operations on the nodes to be installed. Obtain the environmental parameters of each node to be installed; For any node to be installed, based on the environment parameters of the node to be installed, the target computing unit management service installation package corresponding to the node to be installed is selected from the computing unit management service installation package resource pool; the target computing unit management service installation package is sent to the node to be installed; the target computing unit management service installation package is decompressed on the node to be installed, and the target computing unit management service is started. The computing unit management service installation package resource pool includes multiple computing unit management service installation packages adapted to different environmental parameters.

[0005] This application also provides a control device for computing unit management services, including: The module is used to establish passwordless communication between the control node of the target cluster and the node to be installed in the target cluster, and to enable the control node to have the administrator operation privileges of the node to be installed. The deployment module is used to deploy the target runtime environment on the control node, which enables the control node to perform batch operations on the nodes to be installed. The acquisition module is used to obtain the environmental parameters of each node to be installed. The startup module is used to, for any node to be installed, select the target computing unit management service installation package corresponding to the node to be installed from the computing unit management service installation package resource pool based on the environment parameters of the node to be installed; send the target computing unit management service installation package to the node to be installed; decompress the target computing unit management service installation package on the node to be installed; and start the target computing unit management service. The computing unit management service installation package resource pool includes multiple computing unit management service installation packages adapted to different environmental parameters. This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the control method of any of the above-described computing unit management services.

[0006] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the control method for any of the above-described computing unit management services.

[0007] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the control method for any of the above-described computing unit management services.

[0008] This application establishes passwordless communication between the control node of the target cluster and the nodes to be installed in the target cluster, and grants the control node administrator privileges for the nodes to be installed. A target runtime environment is deployed on the control node, enabling the control node to perform batch operations on the nodes to be installed. Environment parameters for each node to be installed are obtained. For any node to be installed, based on its environment parameters, the target computing unit management service installation package corresponding to that node is selected from the computing unit management service installation package resource pool. The target computing unit management service installation package is sent to the node to be installed. The target computing unit management service installation package is decompressed on the node to be installed, and the target computing unit management service is started. This system enables automated and differentiated distribution and installation of computing unit management services, eliminating the need for manual node-by-node installation and configuration. It reduces the deployment time for large-scale cluster computing unit management services from hours to minutes, significantly improving deployment efficiency. Furthermore, by determining the target computing unit management service installation package based on the environment parameters of the nodes to be installed, it resolves deployment failures caused by differences in node environments, greatly reducing the repetitive operation costs for maintenance personnel. Therefore, it addresses the technical problems of low deployment efficiency and high maintenance costs in related technologies, achieving the technical effect of improving the deployment efficiency and reducing the maintenance costs of computing unit management services. Attached Figure Description

[0009] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A schematic diagram of the structure of a control system for computing unit management services provided in this application embodiment; Figure 2 A flowchart illustrating a control method for a computing unit management service provided in this application embodiment; Figure 3 A flowchart illustrating a control method for another computing unit management service provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of a control system for another computing unit management service provided in this application embodiment; Figure 5 A flowchart illustrating the workflow of the distribution and installation module provided in this application embodiment; Figure 6 A schematic diagram of the structure of a control device for managing computing units provided in this application embodiment; Figure 7This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0012] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0013] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] With the rapid development of heterogeneous computing technology, the scale of heterogeneous computing power clusters is constantly expanding. Heterogeneous computing power clusters include artificial intelligence (AI) clusters, and this application describes an AI cluster as an example. As the core computing power support for AI training and inference, the computing power unit's performance and status are crucial for the efficient operation of the cluster. The computing power unit management service, a lightweight computing power unit monitoring tool and system performance monitoring tool developed based on the Go language, is widely used in single-node computing power unit monitoring due to its low resource consumption, read-only operation that does not affect system operation, and support for critical error detection. For example, the computing power unit can be a graphics processing unit (GPU). The computing power unit management service can be GPUD, a tool focused on automating GPU management, designed to simplify monitoring, diagnosis, and maintenance in large-scale GPU environments. It improves the reliability and efficiency of GPU systems through modular devices and low-overhead architecture, and is suitable for AI or machine learning (ML) workload scenarios.

[0015] However, in the context of artificial intelligence clusters, the deployment of computing unit management services relies on manual installation and configuration on each node one by one. This operation is cumbersome and prone to failure due to differences in node environments. As a result, in large-scale clusters (with more than 50 nodes), the deployment efficiency of computing unit management services is extremely low and the operation and maintenance costs are high. Furthermore, manual deployment can lead to problems such as data incompatibility and environmental conflicts.

[0016] To address the aforementioned technical problems, embodiments of this application provide a management and control method and electronic device for a computing unit management service. The management and control method includes: establishing passwordless communication between a control node of a target cluster and nodes to be installed in the target cluster, and granting the control node administrator access to the nodes to be installed; deploying a target runtime environment on the control node, enabling the control node to perform batch operations on the nodes to be installed; acquiring the environment parameters of each node to be installed; for any node to be installed, based on the environment parameters of that node, selecting the target computing unit management service installation package corresponding to that node from a computing unit management service installation package resource pool; sending the target computing unit management service installation package to the node to be installed; decompressing the target computing unit management service installation package on the node to be installed and starting the target computing unit management service; the computing unit management service installation package resource pool includes multiple computing unit management service installation packages adapted to different environment parameters. The method provided by the above solution enables automated and differentiated distribution and installation of the computing unit management service, eliminating the need for manual node-by-node installation and configuration. This reduces the deployment time of the computing unit management service for large-scale clusters from several hours to minutes, improving deployment efficiency. Furthermore, by determining the target computing unit management service installation package based on the environment parameters of the node to be installed, it solves the deployment failure problem caused by differences in node environments, significantly reducing the repetitive operation costs for maintenance personnel. Therefore, it can solve the technical problems of low deployment efficiency and high maintenance costs of computing unit management services in related technologies, achieving the technical effect of improving the deployment efficiency and reducing the maintenance costs of computing unit management services.

[0017] The specific application environment architecture or specific hardware architecture on which the control method of the combined computing unit management service depends is described here.

[0018] The control method for computing unit management services provided in this application embodiment is used to control the computing unit management service, such as... Figure 1The diagram shows the structure of the management system for the computing unit management service upon which this application is based. This system includes a control node for the target cluster and multiple nodes to be installed. The control node establishes passwordless communication with the nodes to be installed in the target cluster and grants the control node administrator privileges for the nodes to be installed. It deploys a target runtime environment, enabling the control node to perform batch operations on the nodes to be installed. It acquires the environment parameters of each node to be installed. For any node to be installed, based on its environment parameters, it selects the corresponding target computing unit management service installation package from the computing unit management service installation package resource pool. The target computing unit management service installation package is then sent to the node to be installed. The target computing unit management service installation package is decompressed on the node to be installed, and the target computing unit management service is started. The computing unit management service installation package resource pool includes multiple installation packages adapted to different environment parameters. The target cluster can be a heterogeneous computing cluster, such as an AI cluster.

[0019] It should be noted that the embodiments in this application use GPU as the computing power unit and GPUD as the computing power unit management service as an example for description.

[0020] Embodiments of this application provide a management and control method for a computing unit management service, applied to the control node of the aforementioned target cluster. Figure 2 A flowchart illustrating the control method for the computing unit management service provided in this application embodiment is shown below. Figure 2 As shown, the control method for this computing unit management service includes the following steps: Step S201: Establish passwordless communication between the control node of the target cluster and the node to be installed in the target cluster, and grant the control node administrator operation privileges for the node to be installed.

[0021] Among them, passwordless communication between the control node and the node to be installed is established based on the Secure Shell (SSH) protocol.

[0022] Step S202: Deploy the target runtime environment on the control node. The target runtime environment enables the control node to perform batch operations on the nodes to be installed.

[0023] In this application, the target operating environment is described as an Ansible environment. Ansible is an automated operation and maintenance tool developed based on Python. It communicates with remote hosts using the SSH protocol and does not require the installation of client or agent programs on the managed end. Its core functions include batch system configuration, program deployment, command execution, and multi-task orchestration. It supports complex operation and maintenance operations through modular design and Playbooks (automation scripts).

[0024] It should be noted that the target runtime environment can also be a SaltStack environment, a Puppet environment, or an environment based on the K8sOperator distribution method, etc.

[0025] Step S203: Obtain the environmental parameters of each node to be installed.

[0026] The environmental parameters include operating system type, operating system version, computing unit driver version, and processor architecture.

[0027] It's important to note that an environment probing playbook is written on the Ansible control node. This playbook uses Ansible's setup module (fact collection module) in conjunction with custom commands to remotely obtain the environment parameters of each node to be installed. Specifically: the `cat / etc / os-release` command obtains the operating system type and version; the `GPU driver version` command obtains the GPU driver version; and the `uname -m` command obtains the Central Processing Unit (CPU) architecture. The CPU architecture refers to the central processing unit architecture. The Ansible control node is the control node of the target cluster.

[0028] It should be further noted that when executing the environment detection Playbook, the collected environment parameters of each node to be installed are stored in JSON format in the node_env_info.json file of the control node. An example of the file structure is as follows: { "node-1": { "os": "CentOS 7", "gpu_driver": "470.103.01", "cpu_arch": "x86_64 }, "node-2": { "os": "Ubuntu 20.04", "gpu_driver": "535.86.10", "cpu_arch": "x86_64 } } Step S204: For any node to be installed, based on the environment parameters of the node to be installed, select the target computing unit management service installation package corresponding to the node to be installed from the computing unit management service installation package resource pool; send the target computing unit management service installation package to the node to be installed; decompress the target computing unit management service installation package on the node to be installed and start the target computing unit management service.

[0029] The computing unit management service installation package resource pool includes multiple computing unit management service installation packages adapted to different environment parameters; the computing unit management service installation package is a self-contained binary installation package.

[0030] Specifically, the Ansible control node writes and distributes the installation package Playbook, reads the environment parameters of each node to be installed from the node_env_info.json file, and matches the corresponding GPUD installation package for each node to be installed from the computing unit management service installation package resource pool based on conditional judgment.

[0031] It should be noted that a resource pool of computing unit management service installation packages is pre-prepared on the control node. This pool includes multiple installation packages adapted to different environment parameters, covering mainstream operating system types and versions (such as CentOS 7 / 8, Ubuntu 20.04 / 22.04, etc.), different GPU driver versions (such as 470.x, 510.x, 535.x, etc.), and self-contained binary installation packages corresponding to different CPU architectures (such as x86_64, ARM64, etc.). Within this resource pool, the computing unit management service installation packages are stored according to the naming convention of "operating system type and version - GPU driver version - CPU architecture" (e.g., gpud_centos7_driver470_x86_64.tar.gz).

[0032] Using Ansible's copy module, the corresponding GPUD installation package is distributed to the / usr / local / bin directory on each node, with file permissions set to 755. It's important to note that 755 is the numerical representation of Linux file permissions, meaning the file / directory owner has read, write, and execute permissions; the group owner and all other users have read and execute permissions, but no write permissions.

[0033] It should be noted that Ansible's breakpoint resume function is enabled during the process of distributing the GPUD installation package matching each node to each node to improve the distribution efficiency of large-scale clusters (use_ssh_args: yes + ssh_args: "-o ControlMaster=auto -o ControlPersist=60s").

[0034] Add installation execution logic to the Playbook distribution package, and remotely execute the following command on each node to be installed using Ansible's command module: Unzip the GPUD installation package: tar -zxvf / usr / local / bin / gpud_ .tar.gz -C / usr / local / bin; Start the GPUD service: gpud up –daemon (runs in background daemon mode).

[0035] The management method for computing unit management services provided in this application embodiment achieves automated and differentiated distribution and installation of computing unit management services through the Ansible tool. It eliminates the need for manual node-by-node installation and configuration, reducing the deployment time of computing unit management services for large-scale clusters from hours to minutes, thus improving deployment efficiency. Simultaneously, it accurately adapts the target computing unit management service installation package based on the environmental parameters of the node to be installed, resolving deployment failures caused by differences in node environments and significantly reducing the repetitive operation costs for maintenance personnel. Therefore, it can solve the technical problems of low deployment efficiency and high maintenance costs of computing unit management services in related technologies, achieving the technical effect of improving the deployment efficiency of computing unit management services and reducing their maintenance costs.

[0036] Embodiments of this application provide a management and control method for a computing unit management service, applied to the control node of the aforementioned target cluster. Figure 3 A flowchart illustrating the control method for the computing unit management service provided in this application embodiment is shown below. Figure 3 As shown, the control method for this computing unit management service includes the following steps: Step S301: Establish passwordless communication between the control node of the target cluster and the node to be installed in the target cluster, and grant the control node administrator privileges for the node to be installed. For details, please refer to [link to relevant documentation]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0037] Step S302: Deploy the target runtime environment on the control node. The target runtime environment enables the control node to perform batch operations on the nodes to be installed. For details, please refer to [link to relevant documentation]. Figure 2Step S202 of the illustrated embodiment will not be described again here.

[0038] Step S303: Obtain the environment parameters for each node to be installed. For details, please refer to [link to relevant documentation]. Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0039] Step S304: For any node to be installed, based on the environment parameters of the node, select the target computing unit management service installation package corresponding to the node from the computing unit management service installation package resource pool; send the target computing unit management service installation package to the node to be installed; decompress the target computing unit management service installation package on the node to be installed, and start the target computing unit management service. The computing unit management service installation package resource pool includes multiple computing unit management service installation packages adapted to different environment parameters; the computing unit management service installation package is a self-contained binary installation package.

[0040] Specifically, step S304 includes: Step S3041: For any node to be installed, based on the operating system type and version of the node to be installed, select the first computing unit management service installation package from the computing unit management service installation package resource pool.

[0041] Step S3042: Based on the computing unit driver version of the node to be installed, select the second computing unit management service installation package from the first computing unit management service installation package.

[0042] Step S3043: Based on the processor architecture of the node to be installed, select the target computing unit management service installation package corresponding to the node to be installed from the second computing unit management service installation package.

[0043] It is understandable that matching the corresponding GPUD installation package from the computing unit management service installation package resource pool for each node to be installed through conditional judgment is specifically achieved through steps S3041 to S3043.

[0044] The control method for computing unit management services provided in this application significantly improves the success rate, efficiency, and reliability of automated deployment of computing unit management services in large-scale, heterogeneous AI clusters through a step-by-step filtering matching logic, and greatly reduces installation failures and subsequent operation and maintenance costs caused by environment incompatibility.

[0045] In some optional implementations, the control method for the above-mentioned computing unit management service further includes: Step a1: For any node to be installed, after starting the corresponding target computing unit management service on the node to be installed, use the status query command to query the running status of the target computing unit management service.

[0046] The status query command is gpud status.

[0047] Step a2: If the target computing unit management service is running, then the target computing unit management service is determined to have started successfully.

[0048] Step a3: If the target computing unit management service is not running, then re-execute the step of starting the corresponding target computing unit management service on the node to be installed.

[0049] Step a4: If the number of times the step of re-executing the corresponding target computing unit management service in the node to be installed exceeds the preset threshold, and the target computing unit management service still fails to start successfully, an alarm will be issued.

[0050] The preset number of attempts threshold is set by technical personnel and is not specifically limited here.

[0051] It should be noted that, for any node to be installed, after starting the corresponding target computing unit management service on that node, execute the Playbook verification. Use Ansible's shell module to remotely execute the GPUD version number check command (gpud version command) to verify whether the GPUD on the node to be installed is installed normally.

[0052] If the GPUD version number can be found, it confirms that the GPUD is installed correctly on the node to be installed. If the GPUD version number cannot be found, the steps to start the corresponding target computing unit management service on the node to be installed are repeated. If the number of times the steps to start the corresponding target computing unit management service on the node to be installed is repeated exceeds a preset threshold, and the target computing unit management service still fails to install correctly, an alarm is issued.

[0053] The control method for the computing unit management service provided in this application embodiment achieves fully automated, multi-round verification and fault handling of the computing unit management service startup status, significantly improving the reliability and maintainability of cluster deployment. Through automatic retries, it can cover most non-fatal transient failures, reducing the need for manual intervention due to accidental failures that interrupt the entire automated process, thus enabling smoother large-scale batch deployments. It fully automates the work that previously required manually logging into each node, checking the service status one by one, and manually handling startup failures. Operations personnel only need to focus on the final alarm information, thereby concentrating their efforts on handling real problems that the automated mechanism cannot solve.

[0054] In some optional implementations, the control method for the above-mentioned computing unit management service further includes: Step b1: Compile and package the source code of the target input plugin into the monitoring data acquisition tool installation package to obtain the target monitoring data acquisition tool installation package.

[0055] In related technologies, there is a lack of dedicated solutions for collecting cluster-level GPU monitoring data. Most methods rely on general commands such as the telegraf exec plugin to collect GPU metrics, which cannot adapt to the proprietary data output format of GPUD. This results in low collection accuracy, data loss, and inadequate error handling mechanisms, and fails to fully utilize the native error detection capabilities of GPUD, such as NVML Xid events and Data Center GPU Manager (DCGM) error detection. NVML Xid events are error or warning events occurring within the GPU hardware or driver. Each Xid event has a unique numerical number corresponding to a specific error type. By parsing the Xid number, operations personnel can accurately pinpoint the root cause of GPU failures. telegraf is a plugin-based metric collection, processing, and transmission agent specifically designed for processing time-series data.

[0056] To address the problems existing in related technologies, the Telegraf custom input plugin, i.e., the target input plugin, in this application embodiment enables accurate collection of cluster-level GPUD monitoring data and fully adapts to the native data format and error detection features of GPUD.

[0057] It should be noted that the monitoring data acquisition tool is Telegraf. A target input plugin, developed using the Go programming language, can be named input.gpud and follows the Telegraf plugin development specifications. Its core functionality includes the following: 1. Data format adaptation: The JSON format data (including GPU utilization, memory usage, temperature, power consumption and error events, etc.) output by GPUD through the GPUD metrics query command (gpud metrics command) is mapped to field types supported by Telegraf, such as float64, string, bool, etc.

[0058] 2. Native error detection integration: By calling the GPUD's NVML interface and DCGM client, NVML Xid events (such as Xid13, Xid43, etc.) and DCGM error codes are captured and encapsulated as independent monitoring metrics.

[0059] 3. Acquisition frequency configuration: This refers to the acquisition period configuration. It supports customizing the acquisition interval (default 10 seconds) through the Telegraf configuration file.

[0060] For example, the core code logic of the target input plugin is as follows: func (g GPUDInput) Gather(acc telegraf.Accumulator) error { / / Execute the gpud metrics command to obtain data output, err:= exec.Command("gpud", "metrics", "--json").Output() if err != nil { return fmt.Errorf("failed to get gpud metrics: %v", err) } / / Parse JSON data var metrics map[string]interface{} if err := json.Unmarshal(output,&metrics); err != nil { return fmt.Errorf("failed to parse metrics: %v", err) } / / Encapsulate the metrics and submit them to Telegraf tags := map[string]string{ "node_name": g.NodeName, "gpu_id":metrics["gpu_id"].(string), } fields := map[string]interface{}{ "utilization_gpu": metrics["utilization_gpu"].(float64), "memory_used":metrics["memory_used"].(float64), "temperature":metrics["temperature"].(float64), "xid_error":metrics["xid_error"].(string), "dcgm_error":metrics["dcgm_error"].(int), } acc.AddFields("gpud_metrics", fields, tags) return nil } Step b2: For any node to be installed, after successfully starting the corresponding target computing unit management service on the node to be installed, install the target monitoring data acquisition tool on the node to be installed based on the target monitoring data acquisition tool installation package, and modify the configuration file of the target monitoring data acquisition tool to add the configuration information of the target input plugin.

[0061] Based on the core code logic of the target input plugin described above, the source code of the target input plugin is determined. This source code is then compiled and packaged into a Telegraf installation package. Telegraf is then installed on each node in the target cluster. The Telegraf configuration file ` / etc / telegraf / telegraf.conf` is modified to add the plugin configuration. [[inputs.gpud]] ## Data Acquisition Interval (Unit: seconds) interval = "10s" ## Timeout timeout = "5s" ## Ignore temporary errors (such as during GPUD service restarts) ignore_transient_errors = true In other words, the configuration information of the target input plugin includes the acquisition interval, timeout period, and configuration to ignore temporary errors.

[0062] Step b3: Start the target monitoring data acquisition tool on the node to be installed. Based on the target input plugin in the target monitoring data acquisition tool, obtain the monitoring index information and error events of the computing power unit in the node to be installed from the target computing power unit management service corresponding to the node to be installed, and submit the monitoring index information and error events of the computing power unit to the target monitoring data acquisition tool.

[0063] Understandably, the target computing unit management service corresponding to the node to be installed can directly obtain monitoring metrics information of the computing units in the node to be installed. These metrics include GPU utilization, memory usage, temperature, and power consumption. The target computing unit management service for the node to be installed captures error events of the computing units in the node to be installed by calling its management library interface and computing unit manager client. The management library interface is the NVML interface, and the computing unit manager client is the DCGM client. Error events include NVML Xid events and DCGM error codes.

[0064] The target input plugin automatically executes the gpud metrics command according to the collection interval specified in the target input plugin configuration information in telegraf, collecting monitoring metrics information and error events of the GPUs on each node to be installed.

[0065] The control method for the computing unit management service provided in this application perfectly adapts to the GPUD proprietary JSON data format through the telegraf-GPUD custom input plugin, avoiding the data format incompatibility problem of general acquisition schemes in related technologies, and effectively improving the acquisition accuracy; it integrates the GPUD native NVML Xid event and DCGM error detection capabilities to achieve full capture of error events, solving the defect of related technologies that cannot utilize the GPUD native error detection function, and ensuring data integrity.

[0066] In some alternative implementations, step b3 above includes: Step b31: Determine the acquisition cycle of the target input plugin based on the configuration information of the target input plugin.

[0067] Step b32: Based on the collection cycle of the target input plugin, use the target input plugin to obtain the monitoring index information and error events of the computing power unit in the target computing power unit management service corresponding to the target computing power unit to be installed.

[0068] The control method for the computing unit management service provided in this application embodiment realizes efficient, customizable, and automated periodic collection of cluster computing unit monitoring data, ensuring the real-time, consistency, and integrity of the monitoring data, and providing a reliable data stream for cluster-level monitoring and intelligent analysis.

[0069] In some optional implementations, the control method for the above-mentioned computing unit management service further includes: In step c1, if the execution time of the target step exceeds the preset time and the target input plugin still fails to obtain the monitoring index information and error events of the computing power unit in the node to be installed that needs to be collected in this collection cycle, then the collection is determined to have failed and a failure log is recorded.

[0070] The preset duration is the timeout period, as mentioned above, and the timeout period can be 5 seconds.

[0071] The target step is to use the target input plugin to obtain the monitoring index information and error events of the computing power unit collected during the current collection period from the target computing power unit management service corresponding to the node to be installed.

[0072] It should be noted that after logging the failure, wait to trigger the next data collection.

[0073] Furthermore, if the execution time of the target step exceeds the preset time, and the target input plugin still has not obtained the monitoring index information and error events of the computing power unit in the node to be installed that needs to be collected in this collection cycle, the running status of the target computing power unit management service is queried using the status query command. If the running status of the target computing power unit management service is not running, the restart command (gpud up) is automatically executed to restart the target computing power unit management service, waiting to trigger the next collection.

[0074] The control method for the computing unit management service provided in this application ensures the real-time performance and integrity of monitoring data through error handling mechanisms such as timeout retries and automatic service restarts.

[0075] In some alternative implementations, step b3 above includes: Step d1: Based on the target input plugin, perform data format conversion processing on the monitoring index information and error events of the computing power unit. The monitoring index information and error events of the computing power unit after data format conversion are adapted to the standard data format of the target monitoring data acquisition tool.

[0076] It should be noted that if the data format conversion of the monitoring indicators and error events of the computing power unit fails, the monitoring indicators and error events of the computing power unit collected in this instance will be ignored.

[0077] Step d2 involves submitting the monitoring indicators and error events of the computing unit after data format conversion to the target monitoring data acquisition tool.

[0078] The control method for the computing unit management service provided in this application embodiment realizes the reliable conversion and submission of monitoring data from the proprietary format of the source to the standard format of the acquisition tool, ensuring the integrity and consistency of subsequent data integration and analysis.

[0079] In some optional implementations, the control method for the above-mentioned computing unit management service further includes: Step e1: Configure the output plugin of the target monitoring data acquisition tool so that the target monitoring data acquisition tool can send the monitoring index information and error events of the computing power unit to the time series database of the control node.

[0080] The time-series database includes a master indicator table and an error event table. The master indicator table stores monitoring indicator information of the computing power unit, and the error event table stores error events of the computing power unit.

[0081] The main indicator table uses node name, computing unit number, target cluster identifier, and rack identifier as index tags, and includes various monitoring indicator fields and collection timestamp fields for the computing unit.

[0082] The error event table uses node name, computing unit number, target cluster identifier, and rack identifier as index labels, and includes error event fields and collection timestamp fields for computing units.

[0083] In related technologies, GPUD monitoring data from each node is scattered and lacks a targeted integration mechanism, making it impossible to achieve cross-node data correlation analysis. This makes it difficult to support cluster-level status assessment and fault location. Furthermore, cluster monitoring solutions in related technologies mostly focus on general system indicators and do not combine the centralized monitoring characteristics of GPUD to achieve in-depth analysis.

[0084] This application embodiment uses a time-series database to associate and structure the monitoring index information and error events of the computing units of each node to be installed, providing a unified data foundation for cluster-level analysis.

[0085] The time-series database can be InfluxDB, adapted to the time-series characteristics of monitoring data. Create a time-series database named gpud_cluster and design the data storage model: Tables (Measurement): gpud_metrics (main metrics table), gpud_errors (error event table).

[0086] Tags: node_name (node ​​name), gpu_id (GPU ID), cluster_id (cluster ID), rack_id (rack ID). The cluster ID is the cluster identifier, and the rack ID is the rack identifier.

[0087] Fields: Various monitoring metric fields, error event fields, and data collection timestamp fields. Monitoring metric fields include GPU utilization, video memory usage, temperature, and power consumption. Error event fields include NVML Xid event fields and DCGM error code fields.

[0088] Specifically, the labels and fields of each table are described above and will not be repeated here.

[0089] It should be noted that the output plugin can be the output.influxdb plugin, which pushes the collected monitoring metrics and error events of the computing units to the InfluxDB database in real time.

[0090] The control method for computing unit management services provided in this application solves the problems of data dispersion and inability to support cluster-level evaluation in traditional solutions by constructing a multi-dimensional data association model to achieve efficient integration and association analysis of cross-node data.

[0091] In some optional implementations, the control method for the above-mentioned computing unit management service further includes: Step f1 involves deduplicating the monitoring metrics and error events of computing units based on node name, computing unit number, target cluster identifier, rack identifier, and data collection timestamp. For computing units with the same target cluster identifier, rack identifier, node name, computing unit number, and data collection timestamp, the first valid data entry is retained.

[0092] Step f2, based on the 3σ principle, determines the normal range of monitoring indicators for the computing power unit.

[0093] Step f3: Based on the normal range of the monitoring indicators of the computing power unit, remove outliers from the monitoring indicator information of the computing power unit, or remove outliers from the monitoring indicator information of the computing power unit (such as GPU utilization > 100%, temperature > 120℃, etc.) based on preset monitoring indicator thresholds. Wherein, σ is the standard deviation.

[0094] Step f4: If there are missing values ​​in the monitoring indicators of the computing power unit, determine whether the missing interval of the missing value is greater than a preset missing interval threshold. If it is greater than the preset missing interval threshold, mark the monitoring indicator information of the computing power unit with the missing value as missing data. If it is not greater than the preset missing interval threshold, use a linear interpolation method to fill the missing value.

[0095] Step f5: If there are missing values ​​in the monitoring indicators of the computing power unit, determine whether the missing percentage of the missing value is greater than the preset missing percentage threshold. If it is greater than the preset missing percentage threshold, mark the monitoring indicator information of the computing power unit with the missing value as missing data.

[0096] Step f6 involves storing the monitoring metrics and error information of the cleaned computing units into the cleaned master metric table and the cleaned error event table in the time-series database. It is understood that the index labels and fields of the cleaned master metric table are the same as those of the master metric table, and the index labels and fields of the cleaned error event table are the same as those of the error event table.

[0097] It should be noted that data cleaning is achieved by deploying data cleaning scripts on the control node. Data cleaning includes deduplication, outlier filtering, and missing value completion. Deduplication involves removing duplicates from the monitoring metrics and error events of the computing units based on node name, computing unit number, target cluster identifier, rack identifier, and data collection timestamp. Outlier filtering determines the normal range of the computing unit's monitoring metrics based on the 3σ principle; based on this normal range, outliers are removed from the computing unit's monitoring metrics, or outliers are removed based on preset monitoring metric thresholds. Missing value completion checks if the missing interval of a computing unit's monitoring metrics exceeds a preset missing interval threshold. If it does, the monitoring metrics of the computing unit containing the missing value are marked as missing data. If it does not exceed the preset missing interval threshold, linear interpolation is used to fill in the missing value. If there are missing values ​​in the monitoring indicators of a computing power unit, it is determined whether the missing percentage of the missing value is greater than the preset missing percentage threshold. If it is greater than the preset missing percentage threshold, the monitoring indicator information of the computing power unit with the missing value is marked as missing data.

[0098] The data cleaning script runs once a minute, processing the raw data from the previous minute. The cleaned data is then written to the cleaned master index table and the cleaned error event table.

[0099] In some alternative implementations, the time-series database includes an aggregated data table for storing monitoring metrics and error events of the computing units.

[0100] The aggregated data table uses node name, computing unit number, target cluster identifier, and rack identifier as index tags, and includes monitoring indicator fields, error event fields, and data collection timestamp fields for each computing unit.

[0101] Understandably, node names and computing unit numbers can be used as association keys to integrate multi-dimensional data (monitoring metrics and error events) from the same GPU, forming a node-GPU-metric-event relationship. Cluster-level association information can also be added, obtaining the cluster ID and rack ID of each node through the cluster configuration file, writing the association information into data tags, and supporting data filtering by cluster and rack dimensions.

[0102] It's important to note that InfluxDB's retention policy features allow for different data retention strategies: Raw data (data in the master indicator table and error event table): retained for 30 days to meet short-term fault rollback needs. Cleaned data (data in the cleaned master indicator table and cleaned error event table): retained for 90 days to support mid-term performance analysis. Aggregated data (data in the aggregate data table): permanently retained to support long-term trend analysis.

[0103] The management and control method for computing unit management services provided in this application constructs a multi-dimensional data association model of "cluster-rack-node-GPU-metric-event" to achieve efficient integration and association analysis of cross-node data, solving the problems of data dispersion and inability to support cluster-level evaluation in traditional solutions. Based on the InfluxDB hierarchical storage strategy, it meets the diverse needs of short-term fault backtracking, mid-term performance analysis, and long-term trend statistics. Combined with data cleaning rules such as 3σ outlier filtering and linear interpolation completion, data reliability is improved, providing a high-quality data foundation for cluster status evaluation.

[0104] In some optional implementations, for any node to be installed, before sending the target computing unit management service installation package to the node to be installed, the above-mentioned computing unit management service control method further includes: Based on the target computing unit management service installation package, determine the first hash value of the target computing unit management service installation package.

[0105] The second hash value of the target computing unit management service installation package is obtained from the computing unit management service installation package resource pool. It is understood that the computing unit management service installation package resource pool also includes the second hash values ​​corresponding to multiple computing unit management service installation packages adapted to different environment parameters. The second hash value is generated before the computing unit management service installation package is stored in the computing unit management service installation package resource pool.

[0106] Compare the first hash value and the second hash value.

[0107] If the first hash value and the second hash value are the same, the target computing unit management service installation package will be sent to the node to be installed.

[0108] If the first hash value and the second hash value are inconsistent, an alarm will be issued, the target computing unit management service installation package will be marked as untrusted, and the installation of the target computing unit management service installation package on the node to be installed will be terminated.

[0109] The control method for the computing unit management service provided in this application verifies the integrity of the target computing unit management service installation package before sending it to the node to be installed. This effectively ensures the integrity and reliability of the computing unit management service deployed to the AI ​​cluster, and avoids cluster-level security risks and deployment failures caused by tampering or damage to the installation package from the root.

[0110] The control method for computing unit management services provided in this application is implemented based on a control system for computing unit management services. Figure 4 A schematic diagram of the structure of the control system for the computing unit management service provided in the embodiments of this application is shown below. Figure 4 As shown, the management system of the computing unit management service includes a control node for the target cluster and multiple nodes to be installed. The control node includes a distribution and installation module, a data acquisition module, and a data integration module. The distribution and installation module is used to collect environmental parameters, adapt and distribute installation packages, automate installation, and verify the installation. Environmental parameter collection involves collecting the environmental parameters of each node to be installed. Installation package adaptation and allocation involves selecting the target computing unit management service installation package corresponding to any node to be installed from the computing unit management service installation package resource pool based on the node's environmental parameters; and sending the target computing unit management service installation package to the node. Automated installation involves decompressing the target computing unit management service installation package on the node and starting the target computing unit management service. Installation verification corresponds to steps a1 to a4 mentioned above and will not be elaborated further here.

[0111] The data acquisition module is used to implement data format parsing, native error capture, exception handling, and plugin configuration. Data format parsing corresponds to step d1 mentioned above and will not be repeated here. Native error capture corresponds to step b3 mentioned above and will not be repeated here. Exception handling corresponds to step c1 mentioned above and will not be repeated here. Plugin configuration corresponds to steps b1 and b2 mentioned above and will not be repeated here.

[0112] The data integration module is used to receive data, clean data, correlate and integrate data, and store data in a hierarchical manner. See the corresponding descriptions above for details, which will not be repeated here.

[0113] Each node to be installed includes computing unit hardware, operating system, computing unit driver, target computing unit management service, and target monitoring data acquisition tool.

[0114] To make the workflow for distributing and installing modules clearer, combined with Figure 5 Describe it. Figure 5 A flowchart of the distribution and installation module provided in the embodiments of this application is shown below. Figure 5As shown, the workflow of the distribution and installation module is as follows: Start the cluster distribution and installation task of the computing unit management service; verify the Ansible control node environment, i.e., verify whether the control node has successfully deployed the target runtime environment; check the integrity of the computing unit management service installation package resource pool, i.e., check whether the computing unit management service installation package resource pool includes multiple computing unit management service installation packages adapted to different environment parameters; determine whether it is ready, i.e., determine whether the control node has successfully deployed the target runtime environment and whether the computing unit management service installation package resource pool is complete. If not ready, i.e., the control node has not successfully deployed the target runtime environment, or the computing unit management service installation package resource pool is incomplete, then the process ends. If ready, i.e., the control node has successfully deployed the target runtime environment and the computing unit management service installation package resource pool is complete, then collect the environment parameters of each node to be installed. Perform installation package adaptation and distribution, as detailed in step S304 above, and will not be repeated here. Determine whether the distribution is successful; if not, retry the distribution. If the distribution is successful, perform automated installation and service startup, and verify the installation result, as detailed in the corresponding description above, and will not be repeated here. The distribution and installation task has been completed.

[0115] The management method for computing unit management services provided in this application uses Ansible tools combined with the environment detection Playbook to automatically collect key information such as cluster node operating system, GPU driver, and CPU architecture. Through the precise matching logic of "environment parameters-installation package", it solves the problems of cumbersome manual deployment of GPUD and poor environment adaptability in related technologies, and is especially suitable for large-scale heterogeneous cluster scenarios.

[0116] By using the Telegraf custom input plugin, not only can the direct parsing and mapping of GPUD's proprietary JSON format data be achieved, but the native error detection interface of GPUD can also be deeply integrated. NVML Xid events and DCGM error codes are encapsulated as independent monitoring indicators, realizing the integrated collection of "monitoring indicators + error events", filling the gap in cluster-level GPUD dedicated collection solutions.

[0117] By using a data integration model with "node name + computing unit number" as the core association key, supplemented by association tags such as cluster ID and rack ID, it supports multi-dimensional data filtering and analysis; based on the InfluxDB tiered retention strategy (30 days for raw data, 90 days for cleaned data, and permanent retention for aggregated data), it achieves a balance between data storage efficiency and practicality, solving the problem of storing and reusing massive amounts of monitoring data.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0119] Embodiments of this application also provide a control device for computing unit management services, such as... Figure 6 As shown, the management device for the computing unit management service includes: Module 601 is established to establish passwordless communication between the control node of the target cluster and the node to be installed in the target cluster, and to enable the control node to have the administrator operation privileges of the node to be installed.

[0120] Deployment module 602 is used to deploy the target runtime environment on the control node, which enables the control node to perform batch operations on the nodes to be installed.

[0121] The acquisition module 603 is used to acquire the environmental parameters of each node to be installed.

[0122] The startup module 604 is used to, for any node to be installed, select the target computing unit management service installation package corresponding to the node to be installed from the computing unit management service installation package resource pool based on the environment parameters of the node to be installed; send the target computing unit management service installation package to the node to be installed; decompress the target computing unit management service installation package on the node to be installed; and start the target computing unit management service.

[0123] The computing unit management service installation package resource pool includes multiple computing unit management service installation packages adapted to different environmental parameters.

[0124] In some alternative implementations, the startup module 604 includes: The first filtering unit is used to select the first computing unit management service installation package from the computing unit management service installation package resource pool for any node to be installed, based on the operating system type and operating system version of the node to be installed.

[0125] The second filtering unit is used to filter out the second computing unit management service installation package from the first computing unit management service installation package based on the computing unit driver version of the node to be installed.

[0126] The third filtering unit is used to filter out the target computing unit management service installation package corresponding to the node to be installed from the second computing unit management service installation package based on the processor architecture of the node to be installed.

[0127] In some optional implementations, the management device for the computing unit management service further includes: The query unit is used to query the running status of the target computing unit management service after starting the corresponding target computing unit management service on any node to be installed, using the status query command.

[0128] The determination unit is used to determine that the target computing unit management service has started successfully if the running status of the target computing unit management service is "running".

[0129] The re-execution unit is used to re-execute the steps of starting the corresponding target computing unit management service on the node to be installed if the target computing unit management service is not running.

[0130] The alarm unit is used to issue an alarm if the number of times the steps to start the corresponding target computing unit management service on the node to be installed are repeated exceeds a preset threshold, and the target computing unit management service still fails to start successfully.

[0131] In some optional implementations, the management device for the computing unit management service further includes: The first acquisition unit is used to compile and package the source code of the target input plugin into the monitoring data acquisition tool installation package to obtain the target monitoring data acquisition tool installation package.

[0132] The installation unit is used to install the target monitoring data acquisition tool on any node to be installed after the corresponding target computing unit management service is successfully started on the node. Based on the target monitoring data acquisition tool installation package, the installation unit modifies the configuration file of the target monitoring data acquisition tool to add the configuration information of the target input plugin.

[0133] The second acquisition unit is used to start the target monitoring data acquisition tool on the node to be installed, and based on the target input plugin in the target monitoring data acquisition tool, to obtain the monitoring index information and error events of the computing power unit in the node to be installed from the target computing power unit management service corresponding to the node to be installed, and submit the monitoring index information and error events of the computing power unit to the target monitoring data acquisition tool.

[0134] In some optional implementations, the second acquisition unit includes: The first determining unit is used to determine the acquisition cycle of the target input plugin based on the configuration information of the target input plugin.

[0135] The second acquisition subunit is used to acquire monitoring index information and error events of the computing power unit in the target computing power unit management service corresponding to the target computing power unit of the target node to be installed, based on the acquisition cycle of the target input plugin.

[0136] In some optional implementations, the management device for the computing unit management service further includes: The second determining unit is used to determine that the current collection has failed and record the failure log if the execution time of the target step exceeds the preset time and the target input plugin still fails to obtain the monitoring index information and error events of the computing power unit in the node to be installed that needs to be collected in this collection cycle.

[0137] The target step is to use the target input plugin to obtain the monitoring index information and error events of the computing power unit collected during the current collection period from the target computing power unit management service corresponding to the node to be installed.

[0138] In some optional implementations, the second acquisition unit includes: The data format conversion processing unit is used to perform data format conversion processing on the monitoring index information and error events of the computing power unit based on the target input plug-in. The monitoring index information and error events of the computing power unit after data format conversion processing are adapted to the standard data format of the target monitoring data acquisition tool.

[0139] The submission unit is used to submit the monitoring indicator information and error events of the computing power unit after data format conversion to the target monitoring data acquisition tool.

[0140] In some optional implementations, the management device for the computing unit management service further includes: The configuration unit is used to configure the output plugin of the target monitoring data acquisition tool so that the target monitoring data acquisition tool can send the monitoring index information and error events of the computing power unit to the time series database in the control node.

[0141] The time-series database includes a master indicator table and an error event table. The master indicator table stores monitoring indicator information of the computing power unit, and the error event table stores error events of the computing power unit.

[0142] The main indicator table uses node name, computing unit number, target cluster identifier, and rack identifier as index tags, and includes various monitoring indicator fields and collection timestamp fields for the computing unit.

[0143] The error event table uses node name, computing unit number, target cluster identifier, and rack identifier as index labels, and includes error event fields and collection timestamp fields for computing units.

[0144] For a description of the features of the control device for the computing power unit management service in the corresponding embodiment, please refer to the relevant description of the control method for the computing power unit management service in the corresponding embodiment, which will not be repeated here.

[0145] Embodiments of this application also provide an electronic device, such as... Figure 7As shown, it includes a processor 701 and a memory 702, in which a computer program is stored. The processor 701 is configured to run the computer program to perform the steps in any of the above-described embodiments of the control method for managing computing units.

[0146] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the control method for managing computing unit services.

[0147] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0148] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the control method for managing computing unit services.

[0149] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the control method for managing computing unit services.

[0150] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0151] The foregoing has provided a detailed description of the control method and electronic device for a computing unit management service provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A control method for a computing power unit management service, characterized in that, include: Establish passwordless communication between the control node of the target cluster and the node to be installed in the target cluster, and enable the control node to have the administrator operation privileges of the node to be installed. A target runtime environment is deployed on the control node, which enables the control node to perform batch operations on the nodes to be installed. Obtain the environmental parameters of each node to be installed; For any node to be installed, based on the environment parameters of the node to be installed, the target computing unit management service installation package corresponding to the node to be installed is selected from the computing unit management service installation package resource pool; the target computing unit management service installation package is sent to the node to be installed; the target computing unit management service installation package is decompressed on the node to be installed, and the target computing unit management service is started. The computing unit management service installation package resource pool includes multiple computing unit management service installation packages adapted to different environmental parameters.

2. The method according to claim 1, characterized in that, The environmental parameters include operating system type, operating system version, computing unit driver version, and processor architecture; for any node to be installed, based on the environmental parameters of that node, the target computing unit management service installation package corresponding to that node is selected from the computing unit management service installation package resource pool, including: For any node to be installed, the first computing unit management service installation package is selected from the computing unit management service installation package resource pool based on the operating system type and version of the node to be installed. Based on the computing unit driver version of the node to be installed, the second computing unit management service installation package is selected from the first computing unit management service installation package; Based on the processor architecture of the node to be installed, the target computing unit management service installation package corresponding to the node to be installed is selected from the second computing unit management service installation package.

3. The method according to claim 1, characterized in that, The method further includes: For any node to be installed, after starting the corresponding target computing unit management service on the node to be installed, use the status query command to query the running status of the target computing unit management service; If the target computing unit management service is running, then the target computing unit management service is determined to have started successfully. If the target computing unit management service is not running, then the step of starting the corresponding target computing unit management service on the node to be installed is re-executed; If the number of times the step of re-executing the corresponding target computing unit management service in the node to be installed exceeds a preset threshold, and the target computing unit management service still fails to start successfully, an alarm will be issued.

4. The method according to claim 1, characterized in that, The method further includes: Compile and package the source code of the target input plugin into the monitoring data acquisition tool installation package to obtain the target monitoring data acquisition tool installation package; For any node to be installed, after successfully starting the corresponding target computing unit management service on the node to be installed, the target monitoring data acquisition tool is installed on the node to be installed based on the target monitoring data acquisition tool installation package. The configuration file of the target monitoring data acquisition tool is modified to add the configuration information of the target input plugin. The target monitoring data acquisition tool is started on the node to be installed. Based on the target input plugin in the target monitoring data acquisition tool, the monitoring index information and error events of the computing power unit in the node to be installed are obtained from the target computing power unit management service corresponding to the node to be installed. The monitoring index information and error events of the computing power unit are submitted to the target monitoring data acquisition tool.

5. The method according to claim 4, characterized in that, The method of obtaining monitoring index information and error events of computing power units in the target computing power unit management service corresponding to the target computing power unit of the target node to be installed, based on the target input plugin in the target monitoring data acquisition tool, includes: Based on the configuration information of the target input plugin, the acquisition cycle of the target input plugin is determined; Based on the collection cycle of the target input plugin, the monitoring index information and error events of the computing power unit in the node to be installed are obtained from the target computing power unit management service corresponding to the node to be installed using the target input plugin.

6. The method according to claim 5, characterized in that, The method further includes: If the execution time of the target step exceeds the preset time, and the target input plugin still fails to obtain the monitoring index information and error events of the computing power unit in the node to be installed that needs to be collected in this collection cycle, then the collection is determined to have failed and a failure log is recorded. The target step involves using the target input plugin to obtain the monitoring index information and error events of the computing power unit collected during the current collection cycle from the target computing power unit management service corresponding to the node to be installed.

7. The method according to claim 4, characterized in that, The step of submitting the monitoring indicator information and error events of the computing power unit to the target monitoring data acquisition tool includes: Based on the target input plugin, the monitoring index information and error events of the computing power unit are processed for data format conversion, wherein the monitoring index information and error events of the computing power unit after data format conversion are adapted to the standard data format of the target monitoring data acquisition tool. The monitoring indicators and error events of the computing unit, after data format conversion, are submitted to the target monitoring data acquisition tool.

8. The method according to claim 4, characterized in that, The method further includes: Configure the output plugin of the target monitoring data acquisition tool so that the target monitoring data acquisition tool sends the monitoring index information and error events of the computing power unit to the time series database in the control node; The time-series database includes a master indicator table and an error event table. The master indicator table is used to store the monitoring indicator information of the computing power unit, and the error event table is used to store the error events of the computing power unit. The main indicator table uses node name, computing unit number, target cluster identifier, and rack identifier as index tags, and includes each monitoring indicator field and collection timestamp field of the computing unit; The error event table uses node name, computing unit number, target cluster identifier, and rack identifier as index tags, and includes error event field and collection timestamp field for computing unit.

9. The method according to claim 8, characterized in that, The time-series database includes an aggregated data table, which is used to store monitoring indicator information and error events of the computing power unit; The aggregated data table uses node name, computing unit number, target cluster identifier, and rack identifier as index tags, and includes monitoring indicator fields, error event fields, and collection timestamp fields for each computing unit.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the control method for the computing unit management service as described in any one of claims 1 to 9 when executing the computer program.

Images