Server cluster monitoring method, device, system and electronic equipment

Abstract

The application provides a server cluster monitoring method, device, system and electronic equipment. The application installs an agent in batches in each server in the server cluster. After the installation is completed, the abnormal performance is located based on the occupation time length of the server cluster in different model training stages. In the case where it is determined that an abnormality occurs in the internal training stage of the server, a plurality of servers included in the server cluster are controlled to execute a P2P bandwidth test tool to detect the transmission performance between GPUs in the server, and the test results of each server are visually displayed. Based on the test results, an abnormal server and an abnormal GPU transmission link in the abnormal server are determined, and automatic troubleshooting of the abnormal server is realized.

Description

Technical Field

[0001] This application relates to the field of computer performance testing technology, and in particular to server cluster monitoring methods, devices, systems and electronic equipment. Background Technology

[0002] Currently, performance evaluation of server clusters typically involves observing single-node log output to assess training efficiency and performance. However, as the number of nodes in a server cluster increases, manual troubleshooting becomes inefficient and prone to errors when poor-performing nodes exist. For example, if a link failure occurs between two GPUs on a single node, causing a decrease in the node's overall computing performance, identifying the abnormal server and the specific GPU experiencing the link failure within a large-scale cluster is extremely difficult. Summary of the Invention

[0003] In view of this, this application provides a server cluster monitoring method, apparatus, system and electronic equipment to automatically investigate abnormal servers in a server cluster.

[0004] The technical solution provided in this application is as follows:

[0005] According to an embodiment of the first aspect of this application, a server cluster monitoring method is provided, the method being applied to a scheduling platform, the scheduling platform being used to schedule a server cluster; the method includes:

[0006] The system batch controls the installation of agents on each server in the server cluster. After the agents are installed on each server, the system visualizes the time taken by the server cluster in different model training stages. Based on the time taken by the server cluster in different model training stages, the system identifies the stages where performance anomalies occur. The different model training stages include: model parameter loading stage, internal server training stage, inter-server synchronization stage, and model parameter writing stage.

[0007] If an anomaly is determined to have occurred during the training phase within the server based on the duration of the server cluster's usage within the server, then the multiple servers included in the server cluster are controlled to execute a P2P bandwidth testing tool; the duration of the server cluster's usage during the training phase within the server is used to characterize the transmission performance between GPUs within each server in the server cluster.

[0008] The test results of each server are displayed visually. Based on these test results, abnormal servers and abnormal GPU transmission links within abnormal servers are identified, prompting an investigation into the abnormal servers and abnormal GPU transmission links within them.

[0009] According to an embodiment of the second aspect of this application, a server cluster monitoring system is provided, the system comprising:

[0010] A server cluster, comprising multiple servers, wherein the servers are used for model training;

[0011] A scheduling platform for executing the methods described in the first aspect.

[0012] According to an embodiment of a third aspect of this application, a server cluster monitoring device is provided, which is applied to a scheduling platform for scheduling a server cluster; the device includes:

[0013] The positioning unit is used to batch control the installation of agents on each server in the server cluster. After the agents are installed on each server, the unit visualizes the time taken by the server cluster in different model training stages to determine the stage where performance anomalies occur based on the time taken by the server cluster in different model training stages. The different model training stages include: model parameter loading stage, internal server training stage, inter-server synchronization stage, and model parameter writing stage.

[0014] The control unit is configured to control multiple servers in the server cluster to execute a P2P bandwidth testing tool if an anomaly is determined to have occurred during the training phase within the server cluster based on the duration of the training phase within the server cluster; the duration of the training phase within the server cluster is used to characterize the transmission performance between GPUs within each server in the server cluster.

[0015] The display unit is used to visualize the test results of each server, identify abnormal servers and abnormal GPU transmission links in abnormal servers based on the test results, and prompt to investigate the abnormal servers and abnormal GPU transmission links in abnormal servers.

[0016] According to an embodiment of the fourth aspect of this application, an electronic device is provided, comprising: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; the processor is configured to execute the machine-executable instructions to implement the method described in the first aspect.

[0017] As can be seen from the above technical solution, this application installs agents on each server in the server cluster in batches. After installation, it locates performance anomalies based on the time occupied by the server cluster in different model training stages. When it is determined that an anomaly occurred in the training stage within the server, it controls multiple servers in the server cluster to execute P2P bandwidth testing tools to detect the transmission performance between GPUs within the server and visualizes the test results of each server. Based on the test results, it identifies the abnormal server and the abnormal GPU transmission link within the abnormal server, thus realizing automatic troubleshooting of abnormal servers. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the principles of this application.

[0019] Figure 1 This is a flowchart of a server cluster monitoring method provided in an embodiment of this application;

[0020] Figure 2 This is a screenshot of the command line output after the installation of the P2P bandwidth testing toolkit is completed, as provided in an embodiment of this application.

[0021] Figure 3 A schematic diagram illustrating the P2P bandwidth test results of a server provided in an embodiment of this application;

[0022] Figure 4 A schematic diagram illustrating the P2P bandwidth test results of a server cluster provided in an embodiment of this application;

[0023] Figure 5 A schematic diagram of the training logs of the server cluster provided in an embodiment of this application;

[0024] Figure 6 A schematic diagram illustrating the GPU performance metrics monitoring results of a server cluster provided in this embodiment of the application;

[0025] Figure 7 This is a schematic diagram summarizing the monitoring results of the server cluster under different training tasks provided in the embodiments of this application.

[0026] Figure 8 This is a schematic diagram illustrating the division of the server cluster training phase in an embodiment of this application;

[0027] Figure 9 A server cluster monitoring system structure diagram provided in this application embodiment;

[0028] Figure 10 A structural diagram of a server cluster monitoring device provided in an embodiment of this application;

[0029] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0031] Please refer to Figure 1 , Figure 1 A flowchart illustrating the server cluster monitoring method provided in this application embodiment.

[0032] In this embodiment, the method can be applied to a scheduling platform for scheduling a server cluster, which includes multiple servers, each of which includes multiple GPUs. This server cluster can be used to execute model training tasks. In the server cluster, each server can be considered a node; unless otherwise specified, "node" may be used to refer to a server in the server cluster.

[0033] like Figure 1 As shown, the method includes the following steps:

[0034] Step 101: Install the agent on each server in the batch control server cluster. After the agent is installed on each server, visualize the time taken by the server cluster in different model training stages to determine the stage where performance anomalies occur based on the time taken by the server cluster in different model training stages.

[0035] A proxy agent can be installed on any server in the server cluster. In this embodiment, the proxy agent can be installed automatically in batches across the server cluster.

[0036] In this embodiment, after the Agent is installed, in order to monitor the link performance between GPUs in the server cluster, a P2P bandwidth testing toolkit can be deployed on the corresponding server by each Agent. Specifically, the installation path of the P2P bandwidth testing toolkit on each server can be set to a fixed path to achieve automated deployment of the P2P bandwidth testing toolkit. After the P2P bandwidth testing toolkit is compiled, the P2P bandwidth testing tool in the P2P bandwidth testing toolkit can be called to test the link performance between GPUs in the server.

[0037] As an example, the aforementioned P2P bandwidth testing toolkit may be the cuda-samples toolkit, which includes the p2pBandwidthLatencyTest performance benchmark program. This application does not impose any limitations on this.

[0038] Please refer to Figure 2 , Figure 2 This is a screenshot of the command line output after the installation of the P2P bandwidth testing toolkit, as provided in this embodiment of the application.

[0039] like Figure 2 As shown, after the Agent is installed, you can choose to deploy the cuda-samples toolkit automatically. Each node has a fixed installation path. After the cuda-samples toolkit is compiled, you can execute the test program p2pBandwidthLatencyTest in the toolkit.

[0040] Since the server cluster can be used to perform model training tasks, after the Agent is installed on each server, the time taken by the server cluster in different model training stages can be visualized to determine the stage where performance anomalies occur based on the time taken by the server cluster in different model training stages.

[0041] Specifically, in this embodiment, the model training phase can be divided into the model parameter loading phase, the server internal training phase, the server inter-synchronization phase, and the model parameter writing phase.

[0042] Among them, the model parameter loading stage refers to the stage from the start of the training task to the completion of loading the model checkpoint parameters; the server internal training stage refers to the stage in which all GPUs in the server complete the specified computing tasks and achieve data synchronization; the server inter-synchronization stage refers to the stage in which different servers synchronize data; and the model parameter writing stage refers to the stage from the end of iterative training to the completion of writing the model checkpoint parameters.

[0043] As one embodiment, the method for locating and determining the stage where performance anomalies occur based on the time occupied by the server cluster in different model training stages may include:

[0044] When the ratio of the time spent in the model parameter loading phase to the total model training time exceeds a first threshold, the model parameter loading phase is determined to be a phase with performance anomalies; wherein, the performance anomaly in the model parameter loading phase indicates anomalies in the storage read performance of the server cluster. When the ratio of the time spent in the server internal training phase to the total model training time exceeds a second threshold, the server internal training phase is determined to be a phase with performance anomalies; wherein, the performance anomaly in the server internal training phase indicates anomalies in the internal network of the server cluster nodes. When the ratio of the time spent in the inter-server synchronization phase to the total model training time exceeds a third threshold, the inter-server synchronization phase is determined to be a phase with performance anomalies; wherein, the performance anomaly in the inter-server synchronization phase indicates anomalies in the cross-node network performance of the server cluster. When the ratio of the time spent in the model parameter writing phase to the total model training time exceeds a fourth threshold, the model parameter writing phase is determined to be a phase with performance anomalies; wherein, the performance anomaly in the model parameter writing phase indicates anomalies in the storage write performance of the server cluster.

[0045] It is easy to understand that in a normal model training process, the time ratio of each model training stage is relatively fixed. However, when an anomaly occurs during training, the time spent in the abnormal training stage usually increases significantly. Therefore, the stage that may be abnormal can be analyzed based on the proportion of the time spent in each model training stage in the total model training time.

[0046] In fact, this embodiment uses the duration of the model training phase to characterize the performance indicators of the server cluster at each stage. For example, the duration of the model parameter loading phase and the model parameter writing phase can be used to characterize the read and write performance of the storage devices in the server cluster, the duration of the internal training phase can be used to characterize the transmission performance between GPUs within the server, and the duration of the inter-server synchronization phase can be used to characterize the network communication performance between nodes, etc.

[0047] As an example, during the normal execution of a model training task, if the time percentages for the model parameter loading stage, the internal server training stage, the inter-server synchronization stage, and the model parameter writing stage are found to be approximately 5%, 70%, 20%, and 5%, respectively, then an alarm threshold can be set for each stage based on its corresponding time percentage during normal model training. Considering potential fluctuations during training, this alarm threshold can be higher than the time percentage of each stage. For example, a first threshold of 10% could be set for the model parameter loading stage, a second threshold of 85% for the internal server training stage, a third threshold of 30% for the inter-server synchronization stage, and a fourth threshold of 10% for the model parameter writing stage. If the time percentage of any stage during the execution of a model training task by the server cluster exceeds the alarm threshold set for that stage, it can be determined that an anomaly may have occurred in that model training stage.

[0048] In this embodiment, the time spent by the server cluster in different model training stages and the total model training time can be visualized. Alternatively, the ratio of the time spent by the server cluster in different model training stages to the total model training time and the alarm threshold set for each model training stage can be visualized. Furthermore, the time spent or the percentage of time spent in model training stages that exceed the alarm threshold set for that stage can be highlighted (e.g., in red). This allows users to determine the abnormal training stage based on the ratio of the time spent by the server cluster in different model training stages to the total model training time.

[0049] In this embodiment, after identifying the stage where performance anomalies occur, corresponding alarm information can be generated to alert the user to the performance anomaly. For example, if the model parameter loading stage is identified as the stage where performance anomalies occur, since performance anomalies in the model parameter loading stage indicate storage read performance anomalies in the server cluster, a storage read performance anomaly alarm can be generated; if the server internal training stage is identified as the stage where performance anomalies occur, since performance anomalies in the server internal training stage indicate network anomalies within the nodes of the server cluster, a node internal network anomaly alarm can be generated; if the inter-server synchronization stage is identified as the stage where performance anomalies occur, since performance anomalies in the inter-server synchronization stage indicate network anomalies across the server cluster, a cross-node network performance anomaly alarm can be generated; if the model parameter writing stage is identified as the stage where performance anomalies occur, since performance anomalies in the model parameter writing stage indicate storage write performance anomalies in the server cluster, a storage write performance anomaly alarm can be generated.

[0050] This concludes the description of step 101. We will now proceed to step 102.

[0051] Step 102: If an anomaly is determined to have occurred during the training phase within the server based on the duration of the server cluster's usage within the server's internal training phase, then control the multiple servers included in the server cluster to execute the P2P bandwidth testing tool.

[0052] In this embodiment, if it is determined according to step 101 that an anomaly occurred during the server's internal training phase, and the duration of the server cluster's time occupied during this phase is used to characterize the transmission performance between GPUs within each server in the server cluster, it indicates that the transmission link between the GPUs within the server may have experienced an anomaly. At this time, multiple servers included in the server cluster can be controlled to execute a P2P bandwidth testing tool. This can be done by controlling each server in the server cluster to execute a P2P bandwidth test, or by controlling a specific server in the server cluster to execute a P2P bandwidth test; this application does not impose any limitations on this.

[0053] Specifically, since the Agent has automatically installed and compiled the P2P bandwidth testing toolkit on the corresponding server after completing the batch installation in step 101, if an anomaly is determined to have occurred during the training phase inside the server, the Agent installed on multiple servers in the server cluster can be directly controlled to perform P2P bandwidth testing on multiple GPUs in the server cluster based on the P2P bandwidth testing toolkit.

[0054] Here, you can control the agents installed on all servers to perform P2P bandwidth tests on the GPUs on all servers, or you can control the agents installed on specific servers included in the server cluster (such as servers that are prone to anomalies or important servers that require special attention) to perform P2P bandwidth tests on multiple GPUs included in these specific servers based on P2P bandwidth testing tools.

[0055] This concludes the description of step 102. We will now proceed to step 103.

[0056] Step 103: Visualize the test results of each server, and identify abnormal servers and abnormal GPU transmission links in abnormal servers based on the test results, so as to prompt the investigation of abnormal servers and abnormal GPU transmission links in abnormal servers.

[0057] In this embodiment, after performing P2P bandwidth testing on the multiple GPUs included in the multiple servers through step 102, a test result is obtained for each server. The test result may include the transmission bandwidth between every two GPUs in that server.

[0058] Please refer to Figure 3 , Figure 3This is a schematic diagram of the P2P bandwidth test results of a server provided in an embodiment of this application.

[0059] like Figure 3 As shown, the test results include four bandwidth matrix tables, which demonstrate the communication performance of eight GPUs (numbered 0-7) in different P2P modes.

[0060] The test modes from top to bottom are: one-way P2P disabled mode, one-way P2P enabled mode, two-way P2P disabled mode, and two-way P2P enabled mode.

[0061] As you can see, Figure 3 The image shows the bandwidth between every two GPUs across the server's eight GPUs under various test modes.

[0062] After obtaining the test results for the server, the test results can be visualized in the scheduling platform interface, and abnormal links can be automatically detected based on the test results.

[0063] Specifically, the bandwidth baseline value for each server can be determined based on the transmission bandwidth between every two GPUs in each server; if the transmission bandwidth of any GPU in any server is lower than a specified proportion of the bandwidth baseline value set for that server, then the transmission link of that GPU is determined to be an abnormal GPU transmission link, and the server is marked as an abnormal server.

[0064] In this embodiment, the method for determining the bandwidth baseline value of each server based on the transmission bandwidth between every two GPUs in the server can be to take the average value of the transmission bandwidth between every two GPUs as the bandwidth baseline value of the server.

[0065] by Figure 3 Taking the test results as an example, for each test mode, the average value of the transmission bandwidth between every two GPUs measured in that mode can be determined as the bandwidth benchmark value of the server in that test mode.

[0066] It is important to note that in the bandwidth matrix of each test mode, the transmission bandwidth values ​​between the same GPU (e.g., Figure 3 In the one-way P2P disabled mode, the transmission bandwidth between GPU0 and GPU0 (2002.88), the transmission bandwidth between GPU1 and GPU1 (2012.96), etc., are not included in the calculation of the bandwidth baseline value.

[0067] After determining the bandwidth baseline value, transmission bandwidth values ​​that are lower than a specified percentage (e.g., 20%) of the bandwidth baseline value can be identified as outliers, indicating that the GPU transmission link corresponding to the transmission bandwidth is an abnormal GPU transmission link and the server is an abnormal server.

[0068] Once the abnormal GPU transmission link is identified, it can be highlighted on a visualization platform (e.g., marked in red) to alert the user to the abnormal location and facilitate further investigation.

[0069] Furthermore, for each server, one can obtain, as follows: Figure 3 The test results shown can be aggregated and displayed on a visualization platform in this embodiment.

[0070] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the P2P bandwidth test results of the server cluster provided in an embodiment of this application.

[0071] like Figure 4 As shown, the detection results of each server (node) in the server cluster can be displayed on the visualization platform, and servers with abnormalities are highlighted. The figure shows the test results of 16 servers, with abnormal servers 8, 10, and 15 highlighted in red to indicate that there are abnormalities in users of different genders and further investigation is needed.

[0072] This concludes the description of step 103.

[0073] In this embodiment, the scheduling platform can not only detect the transmission links between GPUs on each server in the server cluster, but also detect the training performance indicators of the server cluster and the GPU performance indicators of each server.

[0074] Specifically, the method proposed in this embodiment may further include:

[0075] Control the Agent corresponding to each server to obtain the training logs of the server cluster for model training; the training logs should include at least the average time of a single iteration.

[0076] Based on the average time taken per iteration, training performance metrics are generated. These metrics include at least one of floating-point operations, model computing power utilization, and global training throughput. Floating-point operations are used to characterize the computational complexity of the model, computing power utilization is used to characterize the efficiency of hardware resource utilization during model training, and global training throughput is used to characterize the speed at which the model processes data during training.

[0077] Control the Agent corresponding to each server to collect GPU performance metrics of each server. GPU performance metrics include at least one of GPU power consumption, GPU temperature and stream processor utilization.

[0078] The visualization displays at least one of the training performance metrics and GPU performance metrics for different model training tasks.

[0079] In this embodiment, the training performance metrics of the server cluster can be determined based on the training logs of the server cluster during model training.

[0080] Specifically, the training logs include at least the average time taken per iteration when the server cluster trains the model, and the corresponding training performance metrics are calculated according to relevant formulas.

[0081] For example, the floating-point operations (FLOPs) in a single iteration can be calculated using the following formula:

[0082]

[0083] Where B (Batch Size) represents the batch size, used to characterize the number of data samples (such as text and images) processed by the model simultaneously in each iteration; L (Number of Layers) represents the number of model layers. For the Transformer architecture, L represents the number of stacked Transformer encoder / decoder layers; s (Sequence Length) represents the sequence length, used to characterize the length of a single input sample processed by the model; h (Hidden Dimension Size) represents the hidden layer dimension, used to characterize the dimension of the feature vectors inside the model; v represents a summary term, used to characterize the computational overhead related to the number of layers L introduced by operations such as feedforward neural networks and layer normalization.

[0084] Furthermore, based on the ActualFLOPs of floating-point operations in a single iteration and the average time consumed in a single iteration, the floating-point operations of the server cluster per unit time can be obtained, that is, by dividing the ActualFLOPs of floating-point operations in a single iteration by the average time consumed in a single iteration.

[0085] For example, the Model FLOPs Utilization (MFU) can be calculated using the following formula:

[0086] MFU = Floating-point operations per unit time / Maximum floating-point operation capacity that the server cluster can provide per unit time.

[0087] The floating-point operation volume per unit time can be obtained by averaging the time spent in a single iteration according to the above process. When servers are shipped, manufacturers usually provide the maximum floating-point operation capability under ideal conditions. The maximum floating-point operation capability that a server cluster can provide per unit time is usually obtained by aggregating the floating-point operation capabilities of all servers in the cluster.

[0088] For example, the global throughput during training can be calculated using the following formula:

[0089] Global training throughput = batch size / average time per iteration.

[0090] Among them, the global training throughput measures the number of training samples processed per unit time during model training. The batch size is the same as parameter B in the above formula, which is used to represent the number of samples input to the model in each iteration.

[0091] In this embodiment, the calculation method of each training performance index can be set according to actual needs, and this application does not impose any restrictions on it.

[0092] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the training logs of the server cluster provided in an embodiment of this application.

[0093] like Figure 5 As shown, the training log records the average time taken per iteration.

[0094] In addition, the Agent corresponding to each server can be controlled to collect GPU performance metrics of each server. GPU performance metrics include at least one of GPU power consumption, GPU temperature, and stream processor utilization.

[0095] Please refer to Figure 6 , Figure 6 This is a schematic diagram illustrating the monitoring results of GPU performance metrics for a server cluster provided in an embodiment of this application.

[0096] like Figure 6 As shown, the system can visualize individual GPU performance metrics data, such as GPU core clock frequency, GPU memory clock frequency, GPU compute unit utilization, GPU memory utilization, GPU single-card power consumption, memory temperature, and memory usage, for any GPU in each server. It can also visualize the overall performance of GPUs in a server cluster, such as total GPU utilization, total system power consumption, peak reference value, and average GPU core temperature. This application does not impose any restrictions on the GPU performance metrics displayed or the display format.

[0097] In this embodiment, the above-mentioned indicators corresponding to each task can also be summarized to facilitate horizontal comparison.

[0098] Please refer to Figure 7 , Figure 7 This is a schematic diagram summarizing the monitoring results of the server cluster under different training tasks provided in the embodiments of this application.

[0099] like Figure 7 As shown, the monitoring results of the server cluster under different training tasks include model size, training type, task name, pre-configured model parallel strategy (tensor parallelism TP, data parallelism DP, pipeline parallelism PP, and multi-parallelism MDS), length of a single input sample processed by the model, F floating-point operation volume, parameter calculation, number of GPUs, etc. The summarized monitoring results can intuitively show the performance indicators of the server cluster when executing each model training task.

[0100] This concludes the discussion on... Figure 1 Description of server cluster monitoring methods.

[0101] This application installs a proxy agent on each server in a server cluster in batches. After installation, it locates performance anomalies based on the time occupied by the server cluster in different model training stages. When it is determined that an anomaly occurred in the internal training stage of the server, it controls multiple servers in the server cluster to execute a P2P bandwidth testing tool to detect the transmission performance between GPUs within the server and visualizes the test results of each server. Based on the test results, it identifies the abnormal server and the abnormal GPU transmission link in the abnormal server, thus realizing automatic troubleshooting of abnormal servers.

[0102] The following is combined with Figure 8 This application describes the division of the server cluster training phase.

[0103] Please refer to Figure 8 , Figure 8 This is a schematic diagram illustrating the division of the server cluster training phase in an embodiment of this application.

[0104] like Figure 8 As shown, the server cluster includes 4 servers (nodes), and each server includes 8 GPUs.

[0105] Considering that model training is inseparable from factors such as GPU computing power, storage capacity and performance, and network performance, in this embodiment, the scheduling platform divides the model training process into four stages.

[0106] Phase 1: Model parameter loading phase.

[0107] During this phase, model checkpoints are loaded, transferring the model parameters recorded at the checkpoints from... Figure 8 The data is read from the CX storage (storage device) to the server, and the time t1 is recorded from the start of the task to the completion of the checkpoint loading.

[0108] Phase Two: Internal Server Training Phase.

[0109] During this phase, training is performed within each server, and the internal training time t2 of each server can be recorded. For example, the time it takes for a group of TPs to complete data synchronization. The training time in this phase is usually used to characterize the GPU computing power and the network performance inside the machine.

[0110] Phase 3: Inter-server synchronization phase.

[0111] During this phase, data transmission between servers occurs, and the data synchronization time t3 across servers can be recorded, such as the data synchronization time between P and P. The duration of this phase is usually used to characterize the link transmission performance between servers, i.e., the network performance of network cards and switches.

[0112] Phase 4: Model parameter writing phase.

[0113] During this phase, the new model parameters obtained after training are written to the CX storage (storage device) as checkpoints. The time from the completion of iterative training to the completion of checkpoint writing is recorded as t4.

[0114] In this embodiment, if it is determined that the proportion of t2 in the overall training time is higher than the alarm threshold, then the methods of steps 102 and 103 above are executed to locate the abnormal server and abnormal GPU link, which has been described in detail above and will not be repeated here.

[0115] If the performance anomaly of the server cluster is determined to be a storage read performance anomaly or a storage write performance anomaly, the Agent corresponding to each server is controlled to test the data read and write rates of each server based on the storage performance testing tool, so as to determine the server that has a storage read performance anomaly or a storage write performance anomaly based on the data read and write rate.

[0116] If the performance anomaly of the server cluster is determined to be a cross-node network performance anomaly, the Agent corresponding to each server is controlled to detect the communication links between the servers based on the cross-node bandwidth testing tool, so as to determine the inter-server communication links where the cross-node network performance anomaly occurs based on the data read and write rates.

[0117] In this embodiment, if t1 or t4 is found to account for a relatively long proportion of the overall model training time, it is considered that the storage read performance or storage write performance is poor. At this time, the Agent corresponding to each server can be controlled to test the data read and write rate of each server based on storage performance testing tools. For example, tools such as fio can be used to troubleshoot storage problems.

[0118] If t3 is found to account for a long proportion of the overall model training time, it is considered that the cross-node network performance is abnormal. At this time, the Agent corresponding to each server can be controlled to detect the communication links between the servers based on the cross-node bandwidth testing tool, such as by issuing NCCL TEST test, to determine the inter-server communication links where the cross-node network performance is abnormal.

[0119] This concludes the discussion on... Figure 8 Description of the training phase division in the server cluster.

[0120] Please refer to Figure 9 , Figure 9 This is a structural diagram of a server cluster monitoring system provided in an embodiment of this application.

[0121] like Figure 9 As shown, the system includes:

[0122] A server cluster consists of multiple servers used for model training; each server contains multiple GPUs.

[0123] The scheduling platform is used to execute tasks such as... Figure 1 The method.

[0124] In this embodiment, the scheduling platform may include a display device that can visualize the server cluster monitoring results.

[0125] The process of monitoring the server cluster has been described in detail above and will not be repeated here.

[0126] This concludes the discussion on... Figure 9 Description of the server cluster monitoring system.

[0127] Please refer to Figure 10 , Figure 10 This is a structural diagram of a server cluster monitoring device proposed in an embodiment of this application. The device is applied to a scheduling platform, which is used to schedule the server cluster. Figure 10 As shown, the device may include a positioning unit 1001, a control unit 1002, and a display unit 1003.

[0128] Specifically, the device includes:

[0129] The positioning unit 1001 is used to batch control the installation of agents on each server in the server cluster. After the agents are installed on each server, the time taken by the server cluster in different model training stages is visualized to determine the stage where performance anomalies occur based on the time taken by the server cluster in different model training stages. The different model training stages include: model parameter loading stage, internal server training stage, inter-server synchronization stage, and model parameter writing stage.

[0130] The control unit 1002 is configured to control multiple servers included in the server cluster to execute a P2P bandwidth testing tool if it is determined that an abnormality has occurred in the internal training phase of the server cluster based on the time occupied by the server cluster in the internal training phase of the server; the time occupied by the server cluster in the internal training phase of the server is used to characterize the transmission performance between the GPUs inside each server in the server cluster.

[0131] Display unit 1003 is used to visualize the test results of each server, identify abnormal servers and abnormal GPU transmission links in abnormal servers based on the test results, and prompt to investigate the abnormal servers and abnormal GPU transmission links in abnormal servers.

[0132] Optionally, the positioning unit 1001 is specifically used for:

[0133] When the ratio of the time spent in the model parameter loading phase to the total model training time exceeds a first threshold, the model parameter loading phase is determined to be a phase with performance anomalies; wherein, the performance anomaly in the model parameter loading phase indicates an anomaly in the storage read performance of the server cluster, and the time spent in the model parameter loading phase refers to the time taken from the start of the training task to the completion of loading the model checkpoint parameters.

[0134] When the ratio of the time spent in the internal training phase of the server to the total training time of the model exceeds the second threshold, the internal training phase of the server is determined to be a phase in which performance is abnormal; wherein, the performance abnormality in the internal training phase of the server indicates that the internal network of the server cluster is abnormal, and the time spent in the internal training phase of the server refers to the time taken for all GPUs in the server to complete the specified computing tasks and achieve data synchronization.

[0135] When the ratio of the time spent in the inter-server synchronization phase to the total training time of the model exceeds a third threshold, the inter-server synchronization phase is determined to be a phase with performance abnormality; wherein, the performance abnormality of the inter-server synchronization phase indicates that the cross-node network performance of the server cluster is abnormal, and the time spent in the inter-server synchronization phase refers to the time consumed in synchronizing data between different servers.

[0136] When the ratio of the time taken for the model parameter writing phase to the total training time of the model exceeds the fourth threshold, the model parameter writing phase is determined to be a phase with performance abnormality; wherein, the performance abnormality of the model parameter writing phase indicates that the storage write performance of the server cluster is abnormal, and the time taken for the model parameter writing phase refers to the time consumed from the end of iterative training to the completion of model checkpoint parameter writing.

[0137] And / or, the Agent automatically installs and compiles a P2P bandwidth testing toolkit on the corresponding server, the P2P bandwidth testing toolkit including P2P bandwidth testing tools; the control unit 1002 is specifically used for:

[0138] The Agent installed on multiple servers in the control server cluster performs P2P bandwidth testing on multiple GPUs within the server cluster based on the P2P bandwidth testing tool.

[0139] And / or, the test results include the transmission bandwidth between every two GPUs in each server; the display unit 1003 is specifically used for:

[0140] The bandwidth baseline value for each server is determined based on the transmission bandwidth between every two GPUs in each server.

[0141] If the transmission bandwidth of any GPU in any server is lower than a specified percentage of the bandwidth baseline value set for that server, then the transmission link of that GPU is determined to be an abnormal GPU transmission link, and the server is marked as an abnormal server.

[0142] And / or, the display unit 1003 is also used for:

[0143] Control the Agent corresponding to each server to obtain the training logs of the server cluster for model training; the training logs should include at least the average time of a single iteration.

[0144] Based on the average time taken per iteration, training performance metrics are generated. These metrics include at least one of floating-point operations, model computing power utilization, and global training throughput. Floating-point operations are used to characterize the computational complexity of the model, computing power utilization is used to characterize the efficiency of hardware resource utilization during model training, and global training throughput is used to characterize the speed at which the model processes data during training.

[0145] Control the Agent corresponding to each server to collect GPU performance metrics of each server. GPU performance metrics include at least one of GPU power consumption, GPU temperature and stream processor utilization.

[0146] Visualize at least one of the training performance metrics and GPU performance metrics for different model training tasks;

[0147] And / or, the control unit 1002 is also used for:

[0148] If the performance anomaly of the server cluster is determined to be a storage read performance anomaly or a storage write performance anomaly, the Agent corresponding to each server is controlled to test the data read and write rates of each server based on the storage performance testing tool, so as to determine the server that has a storage read performance anomaly or a storage write performance anomaly based on the data read and write rate.

[0149] If the performance anomaly of the server cluster is determined to be a cross-node network performance anomaly, the Agent corresponding to each server is controlled to detect the communication links between the servers based on the cross-node bandwidth testing tool, so as to determine the inter-server communication links where the cross-node network performance anomaly occurs based on the data read and write rates.

[0150] This concludes the process. Figure 10 Description of the server cluster monitoring device.

[0151] This application also provides embodiments that... Figure 10 Hardware structure description of the illustrated device. This hardware structure is... Figure 11 The structure in the illustrated electronic device. Please refer to [link / reference]. Figure 11 , Figure 11 This is a structural diagram of an electronic device provided in an embodiment of this application. Figure 11 As shown, the hardware structure may include: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method disclosed in the above example of this application.

[0152] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the method disclosed in the above examples of this application.

[0153] For example, the aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For instance, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0154] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A server cluster monitoring method, characterized in that, This method is applied to a scheduling platform, which is used to schedule a server cluster; the method includes: The system batch controls the installation of agents on each server in the server cluster. After the agents are installed on each server, the system visualizes the time taken by the server cluster in different model training stages. Based on the time taken by the server cluster in different model training stages, the system identifies the stages where performance anomalies occur. The different model training stages include: model parameter loading stage, internal server training stage, inter-server synchronization stage, and model parameter writing stage. If an anomaly is determined to have occurred during the training phase within the server based on the duration of the server cluster's usage within the server, then the multiple servers included in the server cluster are controlled to execute a P2P bandwidth testing tool; the duration of the server cluster's usage during the training phase within the server is used to characterize the transmission performance between GPUs within each server in the server cluster. The test results of each server are displayed visually. Based on these test results, abnormal servers and abnormal GPU transmission links within abnormal servers are identified, prompting an investigation into the abnormal servers and abnormal GPU transmission links within them.

2. The method according to claim 1, characterized in that, The method of locating and determining the stage where performance anomalies occur based on the time occupied by the server cluster in different model training stages includes: When the ratio of the time spent in the model parameter loading phase to the total model training time exceeds a first threshold, the model parameter loading phase is determined to be a phase with performance anomalies; wherein, the performance anomaly in the model parameter loading phase indicates an anomaly in the storage read performance of the server cluster, and the time spent in the model parameter loading phase refers to the time taken from the start of the training task to the completion of loading the model checkpoint parameters. When the ratio of the time spent in the internal training phase of the server to the total training time of the model exceeds the second threshold, the internal training phase of the server is determined to be a phase in which performance is abnormal; wherein, the performance abnormality in the internal training phase of the server indicates that the internal network of the server cluster is abnormal, and the time spent in the internal training phase of the server refers to the time taken for all GPUs in the server to complete the specified computing tasks and achieve data synchronization. When the ratio of the time spent in the inter-server synchronization phase to the total training time of the model exceeds a third threshold, the inter-server synchronization phase is determined to be a phase with performance abnormality; wherein, the performance abnormality of the inter-server synchronization phase indicates that the cross-node network performance of the server cluster is abnormal, and the time spent in the inter-server synchronization phase refers to the time consumed in synchronizing data between different servers. When the ratio of the time spent in the model parameter writing phase to the total training time of the model exceeds the fourth threshold, the model parameter writing phase is determined to be a phase with performance abnormality; wherein, the performance abnormality of the model parameter writing phase indicates that the storage write performance of the server cluster is abnormal, and the time spent in the model parameter writing phase refers to the time taken from the end of iterative training to the completion of model checkpoint parameter writing.

3. The method according to claim 1, characterized in that, The Agent automatically installs and compiles a P2P bandwidth testing toolkit on the corresponding server, and the P2P bandwidth testing toolkit includes a P2P bandwidth testing tool. The control server cluster includes multiple servers that execute P2P bandwidth testing tools, including: The agent installed on multiple servers in the control server cluster performs P2P bandwidth testing on multiple GPUs within the server cluster based on the P2P bandwidth testing tool.

4. The method according to claim 1, characterized in that, The test results include the transmission bandwidth between every two GPUs in each server; the determination of abnormal servers and abnormal GPU transmission links within those servers based on the test results includes: The bandwidth baseline value for each server is determined based on the transmission bandwidth between every two GPUs in each server. If the transmission bandwidth of any GPU in any server is lower than a specified percentage of the bandwidth baseline value set for that server, then the transmission link of that GPU is determined to be an abnormal GPU transmission link, and the server is marked as an abnormal server.

5. The method according to claim 1, characterized in that, The method also includes: The agent corresponding to each server is controlled to obtain the training logs of the server cluster for model training; the training logs include at least the average time of a single iteration. Based on the average time consumed per iteration, training performance metrics are generated, including at least one of floating-point operations, model computing power utilization, and global training throughput; wherein, floating-point operations are used to characterize the computational complexity of the model, computing power utilization is used to characterize the efficiency of hardware resource utilization during model training, and global training throughput is used to characterize the speed at which the model processes data during training. Control the Agent corresponding to each server to collect the GPU performance indicators of each server, and the GPU performance indicators include at least one of GPU power consumption, GPU temperature and stream processor utilization. The visualization displays at least one of the training performance metrics and GPU performance metrics for different model training tasks.

6. The method according to claim 2, characterized in that, The method also includes: If the performance anomaly of the server cluster is determined to be a storage read performance anomaly or a storage write performance anomaly, the Agent corresponding to each server is controlled to test the data read and write rates of each server based on the storage performance testing tool, so as to determine the server where the storage read performance anomaly or storage write performance anomaly occurs based on the data read and write rates. If the performance anomaly of the server cluster is determined to be a cross-node network performance anomaly, the Agent corresponding to each server is controlled to detect the communication links between the servers based on the cross-node bandwidth testing tool, so as to determine the inter-server communication links where the cross-node network performance anomaly occurred based on the data read and write rates.

7. A server cluster monitoring system, characterized in that, The system includes: A server cluster, comprising multiple servers, wherein the servers are used for model training; A scheduling platform for performing the method as described in any one of claims 1 to 6.

8. A server cluster monitoring device, characterized in that, This device is used in a scheduling platform, which is used to schedule a server cluster; the device includes: The positioning unit is used to batch control the installation of agents on each server in the server cluster. After the agents are installed on each server, the unit visualizes the time taken by the server cluster in different model training stages to determine the stage where performance anomalies occur based on the time taken by the server cluster in different model training stages. The different model training stages include: model parameter loading stage, internal server training stage, inter-server synchronization stage, and model parameter writing stage. The control unit is configured to control multiple servers in the server cluster to execute a P2P bandwidth testing tool if an anomaly is determined to have occurred during the training phase within the server cluster based on the duration of the training phase within the server cluster; the duration of the training phase within the server cluster is used to characterize the transmission performance between GPUs within each server in the server cluster. The display unit is used to visualize the test results of each server, identify abnormal servers and abnormal GPU transmission links in abnormal servers based on the test results, and prompt to investigate the abnormal servers and abnormal GPU transmission links in abnormal servers.

9. The apparatus according to claim 8, characterized in that, The positioning unit is specifically used for: When the ratio of the time spent in the model parameter loading phase to the total model training time exceeds a first threshold, the performance anomaly of the server cluster is determined to be a storage read performance anomaly, and a storage read performance anomaly alarm is generated; the time spent in the model parameter loading phase refers to the time taken from the start of the training task to the completion of loading the model checkpoint parameters; When the ratio of the time spent in the internal training phase of the server to the total training time of the model exceeds the second threshold, the model parameter loading phase is determined to be a phase with performance abnormality; wherein, the performance abnormality in the model parameter loading phase indicates that the storage read performance of the server cluster is abnormal, and the time spent in the internal training phase of the server refers to the time taken for all GPUs in the server to complete the specified computing tasks and achieve data synchronization. When the ratio of the time spent in the inter-server synchronization phase to the total training time of the model exceeds a third threshold, the internal training phase of the server is determined to be a phase with performance abnormality; wherein, the performance abnormality in the internal training phase of the server indicates that the internal network of the server cluster is abnormal, and the time spent in the inter-server synchronization phase refers to the time consumed in synchronizing data between different servers. When the ratio of the time taken to write the model parameters to the total training time of the model exceeds the fourth threshold, the inter-server synchronization phase is determined to be a phase with performance anomalies; wherein, the performance anomalies in the inter-server synchronization phase indicate cross-node network performance anomalies in the server cluster, and the time taken to write the model parameters refers to the time consumed from the end of iterative training to the completion of writing the model checkpoint parameters. And / or, the Agent automatically installs and compiles a P2P bandwidth testing toolkit on the corresponding server, the P2P bandwidth testing toolkit including P2P bandwidth testing tools; the control unit is specifically used for: The Agent installed on multiple servers in the control server cluster performs P2P bandwidth testing on multiple GPUs within the server cluster based on the P2P bandwidth testing tool. And / or, the test results include the transmission bandwidth between every two GPUs in each server; the display unit is specifically used for: The bandwidth baseline value for each server is determined based on the transmission bandwidth between every two GPUs in each server. If the transmission bandwidth of any GPU in any server is lower than a specified percentage of the bandwidth baseline value set for that server, then the transmission link of that GPU is determined to be an abnormal GPU transmission link, and the server is marked as an abnormal server. And / or, the display unit is further configured to: The agent corresponding to each server is controlled to obtain the training logs of the server cluster for model training; the training logs include at least the average time of a single iteration. Based on the average time consumed per iteration, training performance metrics are generated, including at least one of floating-point operations, model computing power utilization, and global training throughput; wherein, floating-point operations are used to characterize the computational complexity of the model, computing power utilization is used to characterize the efficiency of hardware resource utilization during model training, and global training throughput is used to characterize the speed at which the model processes data during training. Control the Agent corresponding to each server to collect the GPU performance indicators of each server, and the GPU performance indicators include at least one of GPU power consumption, GPU temperature and stream processor utilization. Visualize at least one of the training performance metrics and GPU performance metrics for different model training tasks; And / or, the control unit is further configured to: If the performance anomaly of the server cluster is determined to be a storage read performance anomaly or a storage write performance anomaly, the Agent corresponding to each server is controlled to test the data read and write rates of each server based on the storage performance testing tool, so as to determine the server where the storage read performance anomaly or storage write performance anomaly occurs based on the data read and write rates. If the performance anomaly of the server cluster is determined to be a cross-node network performance anomaly, the Agent corresponding to each server is controlled to detect the communication links between the servers based on the cross-node bandwidth testing tool, so as to determine the inter-server communication links where the cross-node network performance anomaly occurred based on the data read and write rates.

10. An electronic device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method according to any one of claims 1 to 6.

Images