Cluster health check method, server cluster, computer storage medium and device

Abstract

A cluster health check method, a server cluster, a computer storage medium and equipment, the cluster health check method comprises: scheduling a first task container to one or more first server nodes in a server cluster, the first task container is configured to start a second task container, the second task container is configured to perform at least one check on the host of the first server node, and the at least one check comprises checking the GPU state of the host.

Description

Technical Field

[0001] This disclosure relates to, but is not limited to, the field of artificial intelligence technology, and in particular to a cluster health check method, server cluster, computer storage medium, and device. Background Technology

[0002] The emergence of large language models has sparked a new wave of artificial intelligence (AI) development, and AI infrastructure has also gradually evolved accordingly. Large model pre-training has now been successfully implemented on GPU clusters; however, training interruptions or slowdowns caused by hardware or software failures can result in tens of millions of dollars being wasted by businesses. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0004] This disclosure provides a cluster health check method, including: scheduling a first task container to one or more first server nodes in the server cluster, the first task container being configured to start a second task container, the second task container being configured to perform at least one health check on the host machine of the first server node, the at least one health check including a check on the GPU status of the host machine.

[0005] This disclosure also provides a server cluster, including: a server cluster consisting of a container controller and multiple server nodes, the container controller including a memory; and a processor coupled to the memory, the container controller being configured to perform the steps of the cluster health check method as described in any embodiment of this disclosure based on instructions stored in the memory.

[0006] This disclosure also provides a computer storage medium storing a computer program that, when executed by a processor, implements the cluster health check method as described in any embodiment of this disclosure.

[0007] This disclosure also provides a computer device including a memory; and a processor coupled to the memory, the processor being configured to perform steps of a cluster health check method as described in any embodiment of this disclosure based on instructions stored in the memory.

[0008] After reading and understanding the accompanying diagrams and detailed descriptions, other aspects can be understood. Attached Figure Description

[0009] The accompanying drawings are provided to further illustrate the technical solutions of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure. The shapes and sizes of the components in the drawings do not reflect actual proportions and are only intended to illustrate the content of this disclosure.

[0010] Figure 1 This is a schematic diagram illustrating the principle of a traditional cloud computing cluster health check.

[0011] Figure 2 A flowchart illustrating a cluster health check method provided as an exemplary embodiment of this disclosure;

[0012] Figure 3 A schematic diagram illustrating the health check principle of a large model training cluster provided in this embodiment of the disclosure;

[0013] Figure 4 A schematic diagram illustrating the health check principle of another large model training cluster provided in this embodiment of the disclosure;

[0014] Figure 5 This is a schematic diagram of a health check result display page provided in an embodiment of this disclosure;

[0015] Figure 6 This is a schematic diagram of a health check process provided in an embodiment of the present disclosure;

[0016] Figure 7 A schematic diagram of a Dmesg hardware and kernel error checking whitelist provided in this embodiment of the disclosure;

[0017] Figure 8 This is a schematic diagram of an IB hardware status check result provided in an embodiment of the present disclosure;

[0018] Figure 9 This is a schematic diagram of an IB communication bandwidth check result provided in an embodiment of the present disclosure;

[0019] Figure 10 This is a schematic diagram of the structure of a container controller provided as an exemplary embodiment of the present disclosure. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be arbitrarily combined with each other.

[0021] Unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" indicate that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, but do not exclude other elements or objects.

[0022] Unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" indicate that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, but do not exclude other elements or objects.

[0023] Kubernetes (K8S) is a new distributed architecture solution based on container technology, and an open-source container cluster management system. A typical K8S cluster consists of one master node and multiple worker nodes. The master node is the cluster control node of K8S. Every K8S cluster needs one master node to be responsible for the management and control of the entire cluster. Basically, all control commands in K8S are sent to it, and it is responsible for the actual execution process. Worker nodes are the workload nodes in the K8S cluster. Each worker node is assigned some workload by the master node. When a worker node fails, its workload is automatically transferred to other worker nodes by the master node.

[0024] The success of large language models has driven the development of large-scale training clusters. Today, large model (i.e., large language model) pre-training has been successfully implemented on multi-GPU clusters, but training interruptions or slowdowns caused by hardware or software failures can result in tens of millions of dollars of wasted funds for enterprises.

[0025] Cluster health checks are fundamental to ensuring the stable operation of large-scale model training clusters. Their purpose is to promptly identify, repair, or remove abnormal nodes, guaranteeing the normal operation of the cluster and services. Before the rise of large-scale model training clusters, traditional cloud computing cluster health checks primarily used the built-in container liveness detection interface of Kubernetes clusters to monitor whether business containers in the cluster had abnormally stopped, such as... Figure 1As shown, the Master node checks whether the containers on each business server (Worker node) are still running through the liveness probe. The Master node can determine whether the containers are running normally based on the responses from each business server.

[0026] However, in large model training clusters, simply checking the health status at the container level cannot adequately guarantee normal training progress and efficiency. This is because current health checks for large model training clusters face the following challenges:

[0027] (1) Large model training clusters need to perform health checks on the host equipment.

[0028] The server nodes in a large model training cluster are usually GPU servers with many high-performance inter-GPU and inter-machine communication devices and underlying software. Furthermore, a large model training cluster requires hundreds of GPU servers to train collaboratively rather than running independently. This makes the normal operation of a large model training cluster more dependent on the health status of hardware devices such as server GPUs, inter-GPU communication devices, and inter-machine communication devices, as well as underlying communication software. Therefore, a large model training cluster needs to perform health checks on the host equipment.

[0029] (2) Many hardware problems in large model training clusters may slow down training and affect training efficiency.

[0030] Large model training clusters are massive and complex systems. Hardware problems caused by communication equipment malfunctions or other factors do not directly interrupt training, but rather slow it down. Moreover, such malfunctions are often not immediately detected and can only be discovered by users during use, thus severely impacting training efficiency.

[0031] (3) The large model training cluster is restricted by the fact that the GPU cannot be shared between K8S containers, which means that the health check task for GPU hardware cannot share the GPU with the training task, thus making it impossible to deploy the health check task.

[0032] Traditional cloud computing clusters are typically CPU clusters, whose custom health check procedures can be freely scheduled to various worker nodes within the cluster by Kubernetes. However, large model training clusters are typically GPU server clusters, and the current AI data center market is dominated by NVIDIA GPUs. In Kubernetes, when using NVIDIA GPU resources, containers (and Pods) cannot share GPU resources. That is, Kubernetes does not support multiple Kubernetes containers sharing the same GPU resource, which has been a persistent problem when scheduling GPU server resources using Kubernetes. For example, suppose server node 1 includes one GPU (named GPU1). Container A is scheduled to run on GPU1 by the Kubernetes Master node. If container B requires a GPU to start, the Kubernetes Master node will not schedule task B to GPU1; it will only schedule it to a GPU on another server node. If no other GPU resources are available, it will queue up, resulting in the health check task for GPU hardware not sharing a GPU with the training task, thus preventing the health check task from being deployed.

[0033] like Figure 2 As shown, this disclosure provides a cluster health check method, including:

[0034] Step 201: Schedule the first task container to one or more first server nodes in the server cluster. The first task container is configured to start a second task container. The second task container is configured to perform at least one health check on the host machine of the first server node. The at least one health check includes a check on the GPU status of the host machine.

[0035] The cluster health check method of this disclosure, by scheduling a first task container to one or more first server nodes in a server cluster, the first task container is configured to start a second task container, and the second task container is configured to perform at least one health check on the host machine of the first server node. The at least one health check includes a check on the GPU status of the host machine. The third task container and the second task container, which execute preset cluster tasks, can be deployed simultaneously on a server node, and the third task container and the second task container can share GPU resources. This allows the health check task and the training task to be jointly scheduled to the same server node, and the health check task will not affect the normal execution of the training task, thus solving the problem of training interruption or slowdown in large model training.

[0036] In this embodiment, the first task container starts the second task container, which can be achieved using Docker outside of Docker (DooD) technology. The core idea of ​​DooD is to share the host machine's Docker process, rather than running a separate Docker process inside the container. The container does not run a Docker process inside; instead, it communicates with the Docker daemon on the host machine by mounting the host machine's docker.sock file (the Unix domain socket that the Docker daemon listens on by default), avoiding additional resource consumption. The Docker client (cli) inside the container interacts directly with the Docker daemon on the host machine. All created containers run on the host machine, rather than being nested within the current container, reducing storage overhead and management complexity.

[0037] In this embodiment of the disclosure, the second task container can also be referred to as a health check container. By combining K8S and DooD technologies to deploy the second task container, this embodiment of the disclosure can achieve K8S scheduling while the actual startup of the container does not completely depend on K8S. The container can be started like executing the "docker run" command (starting the container by the "docker run" command does not restrict multiple containers from sharing the same GPU resources).

[0038] In this embodiment of the disclosure, in step 202, the second task container can also be started in the following way: directly log in to one or more first server nodes, and start the second task container on one or more first server nodes at regular intervals. However, such a startup method requires the user to log in to the first server node one by one, while by deploying the first task container first through the aforementioned cluster scheduling method, and then starting the second task container from the first task container, the user does not need to log in to the first server node one by one.

[0039] In this embodiment of the disclosure, the server cluster includes multiple server nodes, and at least one server node's host machine includes one or more GPU hardware resources. The first server node can be one of the server nodes in the server cluster that includes one or more GPU hardware resources. For example, in a server cluster with four server nodes, where server node 1 includes one GPU hardware resource, server node 2 includes two GPU hardware resources, server node 3 includes four GPU hardware resources, and server node 4 has no GPU resources, then the first server node can be one or more of server nodes 1 to 3. In this embodiment of the disclosure, the number of server nodes in the server cluster and the number of GPU hardware resources on each server node can be set as needed, and this disclosure does not impose any limitations on this.

[0040] In this embodiment of the disclosure, while scheduling the first task container to one or more first server nodes in the server cluster, the first task container can also be scheduled to a server node in the server cluster that does not have GPU hardware resources. At this time, on the server node without GPU hardware resources, the first task container is configured to start a second task container, and the second task container is configured to perform at least one health check on the host machine of the server node. Since the server node does not have GPU hardware resources, the health check performed on the server node does not include the check of the GPU status of the host machine. For example, it may include hardware status check, communication function check, etc.

[0041] In some exemplary embodiments, before or after step 201, the method further includes:

[0042] The third task container is scheduled to one or more second server nodes in the server cluster. The third task container is configured to execute a preset cluster task. The second server nodes may be the same as or different from the first server nodes.

[0043] In this embodiment of the disclosure, in some use cases, scheduling the first task container to one or more first server nodes in the server cluster can begin with the cluster administrator generating a first task description file containing the first task container and submitting it to the server cluster. The task scheduler in the server cluster then schedules the first task container to one or more first server nodes. When the cluster administrator generates the first task description file, they can specify one or more first server nodes for which health checks need to be performed. For example, the one or more first server nodes can be all server nodes in the server cluster, meaning health checks are performed on the host machines of all server nodes. Alternatively, the one or more first server nodes can be some server nodes in the server cluster, meaning health checks are performed on the host machines of some server nodes. Similarly, scheduling the third task container to one or more second server nodes in the server cluster can begin with the client needing to use the server cluster to execute a preset cluster task generating a second task description file containing the third task container and submitting it to the server cluster. The task scheduler in the server cluster then schedules the third task container to one or more second server nodes. When a customer generates a second task description file, they can use node labels and node affinity rules to specify the server nodes that will run the third task container. Affinity rules allow customers to schedule the third task container to one or more specified second server nodes based on node labels, affinity expressions, or computational requirements (such as CPU and memory requirements). For example, a customer can specify the computational requirements of one or more second server nodes that need to execute a preset cluster task. In this case, the task scheduler in the server cluster can select one or more second server nodes to schedule based on the computational requirements.

[0044] In this embodiment, there may be no association between the first server node and the second server node. The number of nodes of the first server node executing the first task container and the number of nodes of the second server node executing the third task container may be equal or unequal. The node name of the first server node executing the first task container and the node name of the second server node executing the third task container may be the same or different.

[0045] In other use cases, the first task container can be configured not only to start the second task container, but also to execute a preset cluster task. That is, both the task that starts the second task container and the preset cluster task are configured in the first task container. In this case, the server nodes executing the first task container and the server nodes executing the third task container are the same (i.e., the number of nodes is equal and the node names are the same).

[0046] In this embodiment of the disclosure, the preset cluster task may include a training task for a large language model. However, this disclosure does not limit this, and the preset cluster task can be set as needed. For example, the preset cluster task may include at least one of the following: a model training task, a model inference task, an image rendering task, a scientific computing task, a simulation computing task, etc., wherein the model corresponding to the model training task and the model inference task can be any model such as a large language model, a visual model, a speech model, or a reinforcement learning model.

[0047] In some exemplary embodiments, the first task container is specifically configured to start the second task container at regular intervals. The timed startup in this disclosure embodiment can be a timed startup at one or more preset time points or a timed startup at preset time intervals.

[0048] In this embodiment, the first task container can periodically start a second task container within the first task container, thereby enabling the server cluster to periodically perform health checks. The specific time node or time interval for the periodic start can be set by the user as needed, and this disclosure does not impose any restrictions on this. For example, the second task container can be started at 24:00 every day, or at a time interval of 48 hours. By periodically starting the second task container, health checks can be triggered on a scheduled basis without requiring manual triggering by the user each time.

[0049] For example, such as Figure 3 As shown, the server cluster can be a large model training cluster, which includes multiple training server nodes. The K8S Master node schedules the training task container (third task container) to one or more training server nodes in the large model training cluster, and schedules the first task container to one or more training server nodes in the large model training cluster at a preset time node (or at every preset time interval). The first task container starts the second task container (health check container). The second task container performs one or more health checks on the host machine of the training server node and writes the results of the health checks to a preset database server. At least one health check includes a check of the GPU status of the host machine.

[0050] In this embodiment of the disclosure, the first task container can use the cron service to start the second task container at regular intervals. In the Linux operating system, cron is a time-based job scheduler that allows users to schedule the execution of commands or scripts at specific times or at specific time intervals; however, this disclosure does not impose any limitations on this. When the server node is a node of other types of operating systems, the appropriate scheduling service can also be selected to start the second task container at regular intervals based on the type of operating system of the server node.

[0051] In some exemplary embodiments, at least two first server nodes have different health check items, and / or at least two first server nodes have a common health check item, and the thresholds for determining whether a node is healthy or not corresponding to the common health check item are different.

[0052] In this embodiment, the health check items corresponding to multiple first server nodes can be the same or different. When performing the same health check item, the threshold standards used by multiple first server nodes to determine their own health status can be the same or different.

[0053] Because the host configurations of different first server nodes may differ—for example, some first server nodes have InfiniBand (IB) interfaces while others do not (IB technology, along with Ethernet, is a computer network communication standard used for high-performance computing, offering extremely high throughput and extremely low latency; currently, most large-scale AI training cluster servers use IB interfaces for connection); or, different first server nodes may use different GPU models, and different GPU models typically have different performance characteristics—different health check items and / or different threshold standards can be set for different first server nodes.

[0054] In some exemplary embodiments, before performing at least one health check on the host machine of the first server node, the second task container is further configured to: obtain the remaining GPU memory and / or GPU utilization of the host machine, and trigger the step of performing at least one health check on the host machine of the first server node when the remaining GPU memory of the host machine is greater than or equal to a preset first threshold and / or the GPU utilization is less than or equal to a preset second threshold.

[0055] In this embodiment, the second task container can determine whether the host machine's GPU is busy based on the host machine's remaining GPU memory and / or GPU utilization. If the host machine's remaining GPU memory is less than a preset first threshold and / or its GPU utilization is greater than a preset second threshold, then the host machine's GPU is determined to be busy. If a health check is insisted on for the first server node at this time, it may not only affect the execution of preset cluster tasks (such as large model training tasks), but the performance detected at this time may not be the true performance of the first server node. Therefore, in this embodiment, when the remaining GPU memory of a first server node's host machine is less than the preset first threshold and / or its GPU utilization is greater than the preset second threshold, a health check is not performed on the first server node, and the first server node is returned to a busy state. The health check for the first server node can wait until the next timed period arrives.

[0056] In this embodiment of the disclosure, the status of the first server node being busy can be determined by the remaining GPU memory and / or GPU utilization of the host machine. However, this disclosure does not limit this, and in some other exemplary embodiments, the status of the first server node being busy can also be determined by other means.

[0057] In this embodiment of the disclosure, when determining whether the first server node is in a busy state solely based on the remaining GPU memory of the host machine, if the remaining GPU memory of the host machine is less than a preset first threshold (e.g., 5G), the first server node is determined to be in a busy state, and no health check is performed on the first server node; if the remaining GPU memory of the host machine is greater than or equal to the preset first threshold, the first server node is determined not to be in a busy state, and a health check can be performed on the first server node.

[0058] In this embodiment of the disclosure, when determining whether the first server node is in a busy state solely based on the GPU utilization of the host machine, if the GPU utilization of the host machine is greater than a preset second threshold (e.g., 10%), the first server node is determined to be in a busy state, and no health check is performed on the first server node; if the GPU utilization of the host machine is less than or equal to the preset second threshold, the first server node is determined not to be in a busy state, and a health check can be performed on the first server node.

[0059] In this embodiment of the disclosure, when determining whether the first server node is in a busy state by the remaining GPU memory and GPU utilization of the host machine, if the remaining GPU memory of the host machine is less than a preset first threshold (e.g., 5G) and / or the GPU utilization is greater than a preset second threshold (e.g., 10%), the first server node is determined to be in a busy state, and no health check is performed on the first server node; if the remaining GPU memory of the host machine is greater than or equal to the preset first threshold and the GPU utilization is less than or equal to the preset second threshold, the first server node is determined not to be in a busy state, and a health check can be performed on the first server node.

[0060] This embodiment of the disclosure determines whether the first server node is in a busy state based on the remaining GPU memory and / or GPU utilization of the host machine, and then decides whether to perform a health check. It can intelligently select whether to perform a health check based on the GPU load of the first server node, so that the health check task will not affect the normal execution of the training task.

[0061] In some exemplary implementations, health check results include: check failed, node healthy, node unhealthy, node busy.

[0062] In this embodiment of the disclosure, after determining that the first server node is not in a busy state, the second task container can start a subprocess to perform a health check. When the subprocess performing the health check exits abnormally, the health check result of the failed check can be returned; when the process performing the health check program ends normally, the execution result of the health check program can be obtained, and the node's health or unhealthy status can be determined based on the execution result of the health check program.

[0063] In some exemplary embodiments, the second task container is further configured to write the health check results of one or more first server nodes into a preset database server. In this embodiment, the health check results in the database server can be obtained by the application server and displayed to the end user.

[0064] For example, such as Figure 4 As shown, the Cron scheduled task container is the first task container, which uses Linux Cron and DooD technology to periodically start the second task container (health check container). The second task container includes a health check program (Function, func) for performing health checks. After performing the health check, the second task container can store the health check results in a preset database (such as MySQL) server. The web backend server periodically retrieves the health check results from the database server and sends them to the web frontend server. The web frontend server then presents the health check results to the end user through a webpage.

[0065] Figure 5 This is an exemplary web rendering diagram of the present disclosure, such as... Figure 5As shown in this example, the server cluster includes four server nodes: node-1 to node-4. Each server node includes four GPU hardware resources (this is just an example and is not a limitation of this disclosure). Through the web front-end server, each node, the status of the node, and other node-related information can be presented on the web page. When the user hovers the mouse over the status of a node, the page can display the relevant information of the last or most recent health checks in the form of a pop-up window. For example, when the user hovers the mouse over a node in a healthy (OK) or busy (BUSY) state, a pop-up window can appear, displaying the time of the last health check, check logs, and other information. When the user hovers the mouse over a node in an unhealthy (UNHEALTHY) state, a pop-up window can appear, displaying the time of the last health check, check logs, the check items that caused the node to be unhealthy, corresponding suggested actions and / or repair commands, and the results of the repair command execution. When the user hovers the mouse over a check that failed (FAIL) state, a pop-up window can appear, displaying the time of the last health check, check logs, the check items that caused the check to fail, and corresponding suggested actions.

[0066] The health check results of this embodiment can be presented to the end user through a webpage or any other means, and this disclosure does not limit this.

[0067] For example, embodiments of this disclosure can analyze the execution results of a health check procedure using regular expression matching, thereby determining whether a node is healthy or unhealthy based on the execution results of the health check procedure.

[0068] In some exemplary embodiments, the second task container is further configured to: when the health check result contains a preset error type, obtain the suggested operation and / or repair command corresponding to the preset error type, write the corresponding suggested operation into a preset database server and / or execute the repair command.

[0069] In this embodiment of the disclosure, when the health check result contains a preset error type (i.e., the node is determined to be unhealthy), it can detect whether there is a suggested operation and / or repair command corresponding to the preset error type. If there is a corresponding suggested operation, it can be pushed to the user, who can then check or process the node according to the suggested operation. If there is a corresponding repair command, it can be automatically executed to repair the node in a timely manner.

[0070] Figure 6 A flowchart illustrating a health check procedure performed by a second task container as exemplified by this disclosure. In one exemplary embodiment, as... Figure 6As shown, the health check process includes: obtaining the number of GPUs with more than 5GB of remaining video memory and a utilization rate of less than 10% on the local machine; if the number of GPUs meeting the conditions is less than 2, the node is determined to be in a busy state, and the node status is written to the database server; if the number of GPUs meeting the conditions is greater than or equal to 2, the local machine's health check list and threshold configuration are loaded, a shell command is created according to the health check list, a child process is started to execute the shell command, and the process is checked to see if the child process has ended. If it has not ended, the process sleeps for 0.1 seconds and then returns to continue checking if the child process has ended, until the child process is found to have ended. Next, it checks whether the child process terminated normally. If the child process terminated abnormally, it returns a check failure status and writes the status to the database server. If the child process terminated normally, it reads the shell command output, uses regular expressions to match the results of each check item, and checks for anomalies in the matching results. If no anomalies are found, it returns a healthy node status and writes the node status to the database server. If anomalies are found, it checks if there is a corresponding repair command. If a corresponding repair command is found, it executes the repair command, returns an unhealthy node status, and writes the node status to the database server. If no corresponding repair command is found, it directly returns an unhealthy node status and writes the node status to the database server.

[0071] In this embodiment of the disclosure, since the preset health check task includes performance checks between two or more GPUs, when the number of GPUs satisfying "more than 5GB of remaining local video memory and less than 10% utilization" is two or more, it is determined that the node is not in a busy state, and a health check can be performed on the node. In some other exemplary embodiments, if the preset health check task does not check the performance between two or more GPUs, but only checks the performance of a single GPU, then when the number of GPUs satisfying "more than 5GB of remaining local video memory and less than 10% utilization" is one or more, it can be determined that the node is not in a busy state, and a health check can be performed on the GPU of that node.

[0072] In some exemplary embodiments, at least one health check includes: a host machine hardware status check, a communication function check, and a GPU performance check. Specifically, the host machine hardware status check and communication function check can promptly identify whether the hardware and communication functions affecting large model training are functioning correctly, while the GPU performance check can promptly identify whether factors causing slowdowns in large model training are functioning correctly.

[0073] In some exemplary implementations, the hardware status check includes at least one of the following: GPU error correction code check, Dmesg hardware and kernel error check.

[0074] Nvidia-smi is a tool for managing and monitoring NVIDIA GPU devices. It allows users to view GPU status, performance, and driver information. For example, the command "nvidia-smi --query-gpu=index,ecc.errors.uncorrected.volatile.total,ecc.mode.current,ecc.mode.pending --format=csv,noheader,nounits" can be used to obtain the GPU Error Correction Code (ECC) status. ECC is used to check for and, in some cases, correct bit errors that may occur during data transmission and storage due to environmental factors.

[0075] In the results returned by the "Nvidia-smi" command, the number of errors can be determined using regular expression matching. If the number of errors is 0, the node is returned as healthy; otherwise, the node is returned as unhealthy. In this embodiment, the suggested action for an unhealthy node status returned by the GPU error correction code check can be pre-set to restart the server, with no corresponding automatic repair command.

[0076] The `dmesg` command is a powerful tool for diagnosing and resolving hardware and kernel errors in Linux systems. It helps users understand the system kernel's operation by displaying information in the kernel's circular buffer, including the status of various devices, error messages, warnings, etc. For example, the `dmesg` command can be used to retrieve the Linux Dmesg log.

[0077] The results returned by the "dmesg" command can be used to match keywords such as "error," "fail," "critical," and "warning" in the logs. The matched results are then compared to a predefined whitelist to check if the matched logs are included in the whitelist. An example whitelist could be as follows: Figure 7 As shown. If the matched log is in the preset whitelist, the node's healthy status is returned; if the matched log is not in the preset whitelist, the node's unhealthy status is returned and an exception log is output. In this embodiment, the suggested operation corresponding to the node's unhealthy status returned by the Dmesg hardware and kernel error check can be preset to read the relevant logs, with no corresponding automatic repair command.

[0078] In some exemplary implementations, the communication function check includes at least one of the following: InfiniBand (IB) communication function check and NCCL communication function check.

[0079] IB is a high-performance, low-latency interconnect technology designed for data centers, supercomputing, and storage networks, intended to replace traditional Ethernet and Fibre Channel. In embodiments of this disclosure, IB communication function checks may include at least one of the following: IB hardware status checks and IB communication bandwidth checks; however, this disclosure is not limiting in this regard.

[0080] For example, such as Figure 8 As shown, the "ibstatus" command can be used to obtain the IB hardware status. In the result returned by the "ibstatus" command, regular expressions are used to match "state" and "phys state" and determine whether it is "ACTIVE" or "LinkUp". If it is "ACTIVE" or "LinkUp", the node's health status is returned; if it is not "ACTIVE" or "LinkUp", the node's unhealthy status is returned and an exception log is output.

[0081] For example, such as Figure 9 As shown, the "ib_send_bw" command can also be used to test whether the IB communication bandwidth "BWaverage" is greater than or equal to the preset rate threshold. If it is greater than or equal to the preset rate threshold, the node health status is returned; if it is less than the preset rate threshold, the node unhealthy status is returned and an exception log is output.

[0082] In this embodiment of the disclosure, the suggested operation corresponding to the node unhealthy status returned by the IB communication function check can be pre-set to check whether the GPU device is properly connected, and the corresponding automatic repair command can be set to restart the IB subnet manager. For example, the IB subnet manager can be restarted using the command "systemctl restart opensm". The IB subnet manager is a service on a machine with an IB interface, responsible for managing the operation and configuration of the entire IB subnet.

[0083] NCCL (NVIDIA Collective Communications Library) is a high-performance communication library optimized for parallel computing in multi-GPU and multi-node environments, supporting a variety of collective communication operations. nccl-tests is an open-source testing toolset developed by NVIDIA for evaluating and validating the functionality and performance of NCCL.

[0084] For example, the "all_reduce_perf" command in the nccl-tests library can be used to test the All_reduce communication rate of GPUs executing 8M, 16M, and 32M data. The All-Reduce operation reduces (e.g., sums) the data (such as gradients) on multiple nodes and then broadcasts it to all nodes, ensuring that each node holds the same copy of the data.

[0085] In the results returned by the "all_reduce_perf" command, you can use regular expressions to match the value of "Avg busbandwidth" and compare the matched value with the preset bandwidth threshold. If it is greater than or equal to the preset bandwidth threshold, the node's healthy status is returned; if it is less than the preset bandwidth threshold, the node's unhealthy status is returned and an exception log is output.

[0086] In this embodiment of the disclosure, the suggested operation corresponding to the node unhealthy status returned by the NCCL communication function check can be preset to check whether the GPU device is properly connected, and there is no corresponding automatic repair command.

[0087] In some exemplary implementations, GPU performance checks include at least one of the following: GPU bandwidth performance check, GPU matrix multiplication performance check, and GPU memory allocation performance check.

[0088] CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model framework developed by NVIDIA for GPU programming. It allows developers to leverage the parallel computing capabilities of NVIDIA GPUs to write code that runs efficiently on GPUs, thereby accelerating computationally intensive tasks. CUDA Samples is an officially provided collection of code examples for learning and testing CUDA programming. It contains a series of sample codes covering various aspects from basic CUDA core concepts to advanced GPU parallel programming techniques. In this disclosure, GPU performance checks can be performed using commands from the CUDASamples library; however, this disclosure does not limit this scope.

[0089] For example, the "bandwidth_test" command from the NVIDIA CUDA Samples library can be used to test the data transfer bandwidth between the GPU and the host CPU, as well as the copy speed between GPU memory and GPU memory. The "bandwidth_test" command returns results including the data transfer speed from the host CPU to the GPU, the data transfer speed from the GPU back to the host CPU, and the data transfer speed from one memory location on the GPU to another. Data transfer speed thresholds can be preset for these three parameters. The tested data transfer speeds are then compared to these thresholds. If multiple tests show data transfer speeds greater than or equal to the corresponding thresholds, the node is considered healthy. If one or more tests show data transfer speeds less than the corresponding thresholds, the node is considered unhealthy, and an error log is output.

[0090] In this embodiment of the disclosure, the suggested operation corresponding to the node unhealthy status returned by the GPU bandwidth performance check can be preset to check whether the GPU hardware is normal, and there is no corresponding automatic repair command.

[0091] Matrix multiplication (GEMM) is a crucial application, particularly in large language models, where it's a hot topic in attention mechanisms. The NVIDIA official library includes four versions of GEMM in its introduction: matrixMul, matrixMul_nvrtc, matrixMulDrv, and matrixMulDynJIT. For example, the "matrixMult" command from the NVIDIA CUDA Samples library can be used to test GPU matrix multiplication performance. The "matrixMult" command returns results including the performance and execution time of the matrix multiplication. Thresholds for performance and execution time can be preset. The tested performance and execution time are compared to these thresholds. If the performance is higher than the threshold and the execution time is lower, the node is considered healthy. If the performance is lower or the execution time is higher, the node is considered unhealthy, and an exception log is output.

[0092] In this embodiment of the disclosure, the suggested operation corresponding to the node unhealthy status returned by the GPU matrix multiplication performance check can be preset to check whether the GPU hardware is normal, and there is no corresponding automatic repair command.

[0093] In some possible implementations, GPU memory allocation performance checks include:

[0094] Perform a preset number of GPU memory allocation operations and calculate the average time taken to perform the preset number of GPU memory allocation operations. Each GPU memory allocation operation includes: allocating a preset number of GPU memory spaces of a preset type and releasing the allocated GPU memory spaces.

[0095] The calculated average time is compared with a preset time threshold. When the calculated average time is lower than the preset time threshold, the node is returned as healthy. When the calculated average time is higher than the preset time threshold, the node is returned as unhealthy and an exception log is output.

[0096] cudaMalloc is a function in CUDA used for allocating GPU memory. It allocates memory on the GPU for CUDA programs to perform parallel computations. This disclosure uses the cudaMalloc function to perform GPU memory allocation performance checks.

[0097] For example, this disclosure can be implemented by writing a CUDA program to execute cudaMalloc a preset number of times (e.g., 100,000 times), allocating and releasing a preset number of spaces of a preset type (e.g., 10K Float spaces) each time, calculating the average time taken for the preset number of cudaMalloc executions, and determining the GPU memory allocation performance based on the calculated average time. For example, the calculated average time is compared with a preset time threshold. If the calculated average time is lower than the preset time threshold, a healthy node status is returned; if the calculated average time is higher than the preset time threshold, an unhealthy node status is returned and an exception log is output. Before executing 100,000 cudaMalloc executions, 100 cudaMalloc executions can be performed first as a warm-up to prevent memory allocation performance instability caused by data loading time at the beginning.

[0098] In this embodiment of the disclosure, the suggested operation corresponding to the node unhealthy status returned by the GPU memory allocation performance check can be preset to check whether the server node has the nvidia-smi command being executed in a loop, and the corresponding automatic repair command can be to clean up all nvidia-smi commands.

[0099] The `nvidia-smi` command is used to display the current status of all GPUs. The `nvidia-smi` command records the GPU status in real time. To stop this command, you need to manually execute the process kill command. For example, the corresponding automatic repair command could be "ps -ef|grep nvidia-smi|grep -v grep|awk'{print$2}'|xargs kill-9".

[0100] The cluster health check method of this disclosure uses a novel deployment method to deploy health check containers into the cluster. This deployment method can ensure the flexible scheduling and unified management capabilities of the K8S cluster, and break the restriction that K8S containers cannot share GPUs. This enables the health check containers and preset cluster task containers to run simultaneously in the K8S cluster by sharing GPU resources, which greatly improves the effective execution time of the preset cluster tasks.

[0101] This disclosure also provides a server cluster, including: a server cluster consisting of a container controller and multiple server nodes, the container controller including a memory; and a processor coupled to the memory, the processor being configured to perform the steps of the cluster health check method as described in any embodiment of this disclosure based on instructions stored in the memory.

[0102] In some possible implementations, such as Figure 4 As shown, the container controller in this embodiment of the disclosure can be a K8S control node; however, this disclosure does not limit it.

[0103] In some possible implementations, such as Figure 4 As shown, the server cluster also includes: a database server and an application server, wherein:

[0104] The second task container is also configured to write the health check results of one or more first server nodes to the database server;

[0105] The application server is configured to read the health check results from the database server and present them to the end user.

[0106] like Figure 10As shown, in one example, the container controller may include a processor 1010, a memory 1020, a bus system 1030, and a transceiver 1040. The processor 1010, memory 1020, and transceiver 1040 are connected via the bus system 1030. The memory 1020 stores instructions, and the processor 1010 executes the instructions stored in the memory 1020 to control the transceiver 1040 to send and receive signals. Specifically, the processor 1010 schedules a first task container to one or more first server nodes in a server cluster. The first task container is configured to launch a second task container, and the second task container is configured to perform at least one health check on the host machine of the first server node. The at least one health check includes a check of the GPU status of the host machine.

[0107] It should be understood that processor 1010 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0108] Memory 1020 may include read-only memory and random access memory, and provides instructions and data to processor 1010. A portion of memory 1020 may also include non-volatile random access memory. For example, memory 1020 may also store device type information.

[0109] In addition to a data bus, the bus system 1030 may also include a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 10 The general labeled all buses as Bus System 1030.

[0110] In implementation, the processing performed by the processing device can be accomplished through integrated logic circuits in the hardware of the processor 1010 or through software instructions. That is, the method steps of this embodiment can be executed by a hardware processor, or by a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other storage media. This storage medium is located in memory 1020. The processor 1010 reads information from memory 1020 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, further details are omitted here.

[0111] This disclosure also provides a computer storage medium storing a computer program that, when executed by a processor, implements the cluster health check method as described in any embodiment of this disclosure.

[0112] In some possible implementations, various aspects of the cluster health check method provided in this application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps in the cluster health check method according to various exemplary embodiments of this application described above. For example, the computer device can execute the cluster health check method described in the embodiments of this application.

[0113] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0114] This disclosure also provides a computer device, which may include: at least one processor, at least one memory, a power supply, a display screen, an input / output interface, a communication interface, and a communication bus. The memory stores a computer program, which is loaded and executed by the processor to implement the relevant steps in the cluster health check method disclosed in any of the foregoing embodiments.

[0115] In this embodiment, the power supply is used to provide operating voltage for various hardware devices on the computer device; the communication interface can create a data transmission channel between the computer device and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0116] As a carrier for storing resources, a memory can be a read-only memory, a random access memory, a disk, or an optical disk, etc. The resources stored on it can include operating systems, computer programs, etc., and the storage method can be temporary storage or permanent storage.

[0117] The operating system is used to manage and control the various hardware devices and computer programs on the computer device, and may be Windows Server, Netware, Unix, Linux, etc. In addition to computer programs capable of performing the cluster health check method executed by the computer device as disclosed in any of the foregoing embodiments, the computer programs may further include computer programs capable of performing other specific tasks.

[0118] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0119] While the embodiments disclosed herein are as described above, the content is merely for the purpose of facilitating understanding of this disclosure and is not intended to limit the invention. Any person skilled in the art may make any modifications and changes to the form and details of the implementation without departing from the spirit and scope of this disclosure; however, the patent protection scope of this invention shall still be determined by the scope defined in the appended claims.

Claims

1. A cluster health check method, characterized in that, include: A first task container is scheduled to one or more first server nodes in a server cluster. The first task container is configured to launch a second task container. The second task container is configured to perform at least one health check on the host machine of the first server node, the at least one health check including a check on the GPU status of the host machine.

2. The method according to claim 1, characterized in that, The server cluster includes multiple server nodes, and the host of at least one server node includes one or more GPU hardware resources. The first server node is a server node that includes one or more of the said GPU hardware resources.

3. The method according to claim 1, characterized in that, The method further includes: scheduling a third task container to one or more second server nodes in the server cluster, wherein the third task container is configured to execute a preset cluster task, and the second server node is the same as or different from the first server node.

4. The method according to claim 3, characterized in that, The preset cluster tasks include: training large language models.

5. The method according to claim 1, characterized in that, The first task container is specifically configured to start the second task container at regular intervals.

6. The method according to claim 1, characterized in that, Before performing at least one health check on the host machine of the first server node, the second task container is further configured to: obtain the remaining GPU memory and / or GPU utilization of the host machine, and trigger the step of performing at least one health check on the host machine of the first server node when the remaining GPU memory of the host machine is greater than or equal to a preset first threshold and / or the GPU utilization is less than or equal to a preset second threshold.

7. The method according to claim 1, characterized in that, The at least one health check includes: a hardware status check of the host machine, a communication function check of the host machine, and a GPU performance check of the host machine.

8. The method according to claim 7, characterized in that, The hardware status check includes at least one of the following: GPU error correction code check, Dmesg hardware and kernel error check.

9. The method according to claim 7, characterized in that, The communication function check includes at least one of the following: unlimited bandwidth IB communication function check, NCCL communication library communication function check.

10. The method according to claim 7, characterized in that, The GPU performance check includes at least one of the following: GPU bandwidth performance check, GPU matrix multiplication performance check, and GPU memory allocation performance check.

11. The method according to claim 10, characterized in that, The GPU memory allocation performance check includes: Perform a preset number of GPU memory allocation operations and calculate the average time taken to perform the preset number of GPU memory allocation operations. Each GPU memory allocation operation includes: allocating a preset number of GPU memory spaces of a preset type and releasing the allocated GPU memory spaces. The calculated average time is compared with a preset time threshold. When the calculated average time is less than or equal to the preset time threshold, the node is returned as healthy. When the calculated average time is greater than the preset time threshold, the node is returned as unhealthy and an inspection log is output.

12. The method according to claim 1, characterized in that, The second task container is also configured to write the health check results of one or more of the first server nodes into a preset database server.

13. The method according to claim 12, characterized in that, The health check results include: check failed, node healthy, node unhealthy, node busy.

14. The method according to claim 1, characterized in that, The second task container is also configured to: when the health check result contains a preset error type, obtain the suggested operation and / or repair command corresponding to the preset error type, write the suggested operation into a preset database server and / or execute the repair command.

15. The method according to claim 1, characterized in that, At least two of the first server nodes have different health check items, and / or at least two of the first server nodes have a common health check item, and the thresholds for determining whether a node is healthy or not corresponding to the common health check item are different.

16. A server cluster, characterized in that, include: A server cluster comprising a container controller and multiple server nodes, the container controller including a memory; and a processor coupled to the memory, the container controller being configured to perform the steps of the cluster health check method as described in any one of claims 1 to 15 based on instructions stored in the memory.

17. The server cluster according to claim 16, characterized in that, The container controller is a Kubernetes (K8S) control node.

18. The server cluster according to claim 16, characterized in that, Also includes: Database server and application server, of which: The second task container is also configured to write the health check results of one or more of the first server nodes into the database server; The application server is configured to read the health check results from the database server and present them to the end user.

19. A computer storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the cluster health check method as described in any one of claims 1 to 15.

20. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 15.

Images