Performance monitoring system and performance monitoring method

Abstract

The invention relates to the field of model training, and provides a performance monitoring system and a performance monitoring method. The system comprises a computing device cluster and a main control device, each computing device comprises an acquisition server, and the acquisition server is used for acquiring a training performance index of the computing device when the computing device executes a training task and outputting the acquired training performance index to the main control device; the main control equipment is used for receiving and storing the training performance indexes, analyzing the stored training performance indexes, judging whether each computing equipment meets a preset condition or not, obtaining a judgment result, and determining at least one to-be-optimized computing equipment in the computing equipment cluster based on the judgment result; and generating and executing an optimization strategy according to the use conditions of the to-be-optimized computing device and the computing device cluster. The system is suitable for computing device clusters of any scale, and can meet the performance monitoring and analysis tuning requirements of the large-scale computing device clusters in a distributed model training scene.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of model training, and in particular, to a performance monitoring system and a performance monitoring method. BACKGROUND

[0002] A large model refers to a deep learning model with a large number of parameters and a large amount of training data, such as GPT, BERT, etc. With the rapid development of large models, the computing resources required for training large models also grow exponentially, and the computing devices used for training are expanded from tens to tens of thousands, and distributed training of large models is achieved through large-scale computing device clusters.

[0003] In order to monitor the training of large models, index monitoring and performance analysis are needed during the training of large models. Most of the traditional index monitoring and performance analysis schemes are developed for training scenarios with small scale, and are not suitable for such large-scale distributed training scenarios. Therefore, there is an urgent need for a performance monitoring system that is adapted to large-scale computing device clusters to meet the performance monitoring and analysis optimization requirements of large-scale computing device clusters in distributed model training scenarios. SUMMARY

[0004] Therefore, the present disclosure provides a performance monitoring system and a performance monitoring method. The system is adapted to computing device clusters of any scale, can monitor the training performance indicators of computing devices, analyze whether the computing devices need to be optimized, generate corresponding optimization strategies and execute them, and meet the performance monitoring and analysis optimization requirements of large-scale computing device clusters in distributed model training scenarios.

[0005] According to an aspect of the present disclosure, a performance monitoring system is provided, which includes a computing device cluster for executing a training task and a master device. Each computing device includes a collection service end, which is configured to collect training performance indicators of the computing device when the computing device executes a training task, and output the collected training performance indicators to the master device. The master device is configured to receive and store the training performance indicators, analyze the stored training performance indicators to determine whether each computing device meets a preset condition, obtain a determination result, determine at least one to-be-optimized computing device in the computing device cluster based on the determination result, generate an optimization strategy based on the to-be-optimized computing device and the usage of the computing device cluster, and execute the optimization strategy.

[0006] In a possible implementation, the collection service end includes a unified index abstraction layer configured to define at least one index interface common to different training frameworks. The collection service end is specifically configured to collect the training performance indicators of the computing device through the index interface, and each index interface corresponds to the collection of one training performance indicator.

[0007] In one possible implementation, the data collection server is specifically used to collect the training performance indicator once using any indicator interface when the name of any training performance indicator corresponding to an indicator interface appears during the execution of a training task on the computing device.

[0008] In one possible implementation, the data acquisition server integrates at least one indicator plugin. Specifically, the data acquisition server is used to collect training performance indicators of the computing device through the indicator plugin, with each indicator plugin corresponding to the collection of one training performance indicator.

[0009] In one possible implementation, the data collection server is specifically used to collect the training performance indicator once using any indicator plugin when the name of any training performance indicator corresponding to an indicator plugin appears during the execution of a training task on the computing device.

[0010] In one possible implementation, the master control device includes a time series database. Specifically, the master control device is used to: add time series information to the received training performance indicators to obtain updated training performance indicators; store the updated training performance indicators in the time series database; select training performance indicators within a target time period from the time series database; analyze the selected training performance indicators; and determine at least one computing device to be optimized in the computing device cluster.

[0011] In one possible implementation, the main control device is specifically used to: cluster selected training performance indicators of the same type, and obtain a corresponding cluster for each training performance indicator; calculate the threshold corresponding to each training performance indicator based on the cluster corresponding to each training performance indicator; for each computing device, use at least one selected training performance indicator of the computing device and the corresponding threshold to determine whether the computing device meets the preset conditions; when the computing device meets the preset conditions, the computing device is determined as the computing device to be optimized.

[0012] In one possible implementation, the preset conditions include: the difference between the N training performance indicators of the computing device and the corresponding threshold is less than M times the corresponding threshold, where M is greater than 1 and N is a positive integer.

[0013] In one possible implementation, the threshold corresponding to each training performance metric is equal to the mean, mode, median, variance, or standard deviation of the training performance metrics within the corresponding cluster.

[0014] In one possible implementation, when there are idle computing devices in the computing device cluster, the optimization strategy includes using the idle computing devices to perform training tasks instead of the computing device to be optimized; when there are no idle computing devices in the computing device cluster, the optimization strategy includes restarting the computing device to be optimized.

[0015] In one possible implementation, the master control device is further configured to display to the user at least one of the at least one computing device to be optimized, the training performance indicators received by the master control device, and the optimization strategy executed, through a visualization module.

[0016] According to another aspect of this disclosure, a performance monitoring method is provided, the method being applied to a performance monitoring system, the system including a cluster of computing devices for executing training tasks and a main control device, each computing device including a data acquisition server, the method comprising: the data acquisition server collecting training performance indicators of the computing device it is executing a training task, and outputting the collected training performance indicators to the main control device; the main control device receiving and storing the training performance indicators, analyzing the stored training performance indicators to determine whether each computing device meets preset conditions, obtaining a judgment result, and determining at least one computing device in the computing device cluster to be optimized based on the judgment result; generating an optimization strategy based on the usage of the computing device to be optimized and the computing device cluster; and executing the optimization strategy.

[0017] In one possible implementation, the data acquisition server includes a unified metric abstraction layer for defining at least one metric interface common to different training frameworks; the data acquisition of training performance metrics of the computing device includes: acquiring the training performance metrics of the computing device through the metric interface, with each metric interface corresponding to the acquisition of one training performance metric.

[0018] In one possible implementation, collecting the training performance metrics of the computing device through the metric interface includes: when the name of any training performance metric corresponding to any metric interface appears during the execution of the training task on the computing device, using that metric interface to collect the training performance metric once.

[0019] In one possible implementation, the data acquisition server integrates at least one indicator plugin, and the data acquisition of the training performance indicators of the computing device includes: acquiring the training performance indicators of the computing device through the indicator plugin, with each indicator plugin corresponding to the acquisition of one training performance indicator.

[0020] In one possible implementation, the step of collecting training performance metrics of the computing device through the metric plugin includes: when the name of a training performance metric corresponding to any metric plugin appears during the execution of a training task on the computing device, the metric plugin is used to collect the training performance metric once.

[0021] In one possible implementation, the master control device includes a time-series database. Receiving and storing training performance indicators includes: adding time-series information to the received training performance indicators to obtain updated training performance indicators, and storing the updated training performance indicators in the time-series database. Analyzing the stored training performance indicators to determine at least one computing device to be optimized in the computing device cluster includes: selecting training performance indicators within a target time period from the time-series database, analyzing the selected training performance indicators, and determining at least one computing device to be optimized in the computing device cluster.

[0022] In one possible implementation, the step of analyzing the selected training performance metrics to determine at least one computing device to be optimized in the computing device cluster includes: clustering the selected training performance metrics of the same type to obtain a corresponding cluster for each training performance metric; calculating a threshold corresponding to each training performance metric based on the cluster corresponding to each training performance metric; for each computing device, using at least one selected training performance metric of the computing device and the corresponding threshold, determining whether the computing device meets a preset condition; and when the computing device meets the preset condition, determining the computing device as the computing device to be optimized.

[0023] In one possible implementation, the preset conditions include: the difference between the N training performance indicators of the computing device and the corresponding threshold is less than M times the corresponding threshold, where M is greater than 1 and N is a positive integer.

[0024] In one possible implementation, the threshold corresponding to each training performance metric is equal to the mean, mode, median, variance, or standard deviation of the training performance metrics within the corresponding cluster.

[0025] In one possible implementation, when there are idle computing devices in the computing device cluster, the optimization strategy includes using the idle computing devices to perform the training task in place of the computing device to be optimized; when there are no idle computing devices in the computing device cluster, the optimization strategy includes restarting the computing device to be optimized.

[0026] In one possible implementation, the method further includes: the master control device displaying to the user at least one of the at least one computing device to be optimized, the training performance indicators received by the master control device, and the optimization strategy executed, through a visualization module.

[0027] The performance monitoring system according to an embodiment of this disclosure includes a cluster of computing devices for performing training tasks and a main control device. Each computing device includes a data acquisition server, which is used to collect training performance indicators of the computing device when it performs a training task, and output the collected training performance indicators to the main control device. The main control device is used to receive and store the training performance indicators, analyze the stored training performance indicators to determine whether each computing device meets preset conditions, obtain a judgment result, determine at least one computing device in the computing device cluster to be optimized based on the judgment result, generate an optimization strategy based on the computing device to be optimized and the usage of the computing device cluster, and execute the optimization strategy. This method deploys an independent data acquisition server on each computing device, and aggregates the training performance data collected by each data acquisition server through the main control device, realizing real-time collection and uploading of key indicators during the training process; it releases the indicator collection pressure of the main control device to the computing devices, reduces the workload of the main control device, and is suitable for distributed model training scenarios of larger-scale computing device clusters. This system can monitor the training performance indicators of computing devices, analyze whether the computing devices need optimization, generate corresponding optimization strategies and execute them, meet the performance monitoring, analysis and optimization needs of large-scale computing device clusters in distributed model training scenarios, improve overall training efficiency, reduce resource waste and reduce manual intervention debugging costs.

[0028] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0029] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0030] Figure 1 This illustrates an exemplary application scenario of a performance monitoring system according to embodiments of the present disclosure.

[0031] Figure 2 A schematic diagram showing the structure of a performance monitoring system according to an embodiment of the present disclosure is provided.

[0032] Figure 3 A schematic diagram showing the structure of a master control device according to an embodiment of the present disclosure is provided.

[0033] Figure 4 This diagram illustrates clustering of selected training performance metrics according to an embodiment of the present disclosure.

[0034] Figure 5 A schematic diagram illustrating the flow of a performance monitoring method according to an embodiment of the present disclosure is shown. Detailed Implementation

[0035] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0036] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0037] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0038] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0039] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0040] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0041] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0042] Existing technology-based indicator monitoring and performance analysis solutions can be divided into three categories.

[0043] The first category is bottleneck detection and performance optimization based on system resource monitoring. By collecting system resource monitoring indicators at the node (computing device) level (such as CPU / GPU utilization, memory usage, network communication latency, disk communication latency, etc.), the monitored indicators are analyzed to identify computing devices with low performance (i.e., computing devices causing performance bottlenecks), and performance optimization is performed on these low-performance computing devices.

[0044] The disadvantage of this type of solution is:

[0045] 1. The granularity of the indicator collection is too coarse, making it impossible to determine in which training stage (such as the forward propagation stage, back propagation stage, communication stage, etc.) the low-performance computing device is causing the performance bottleneck, and also impossible to judge the performance differences between different training stages.

[0046] 2. System resource monitoring metrics are not directly related to the logic of the large model, and cannot reveal the impact of deep-seated behaviors such as the structure of the large model, parameter distribution, and gradient updates on the training performance of computing devices. Therefore, it is difficult to provide effective optimization strategies based on system resource monitoring metrics.

[0047] 3. Using static thresholds to judge the performance of computing devices can be misled by system load or short-term fluctuations. For example, when collecting network communication latency data for a computing device, fluctuations may result in higher measured values, which could lead to the device being misjudged as having low performance. Optimizing such a device could actually reduce training efficiency.

[0048] The second category is bottleneck detection and performance optimization based on probes. By injecting probes during the training of large models, the execution time and resource usage of each operation (Op) are recorded as indicators. The indicators are analyzed to identify computing devices with low performance, and the indicators and low-performance computing devices can be visualized.

[0049] The disadvantage of this type of solution is:

[0050] 1. It has a high overhead and cannot be deployed on a large scale across all computing devices in a cluster. Otherwise, it will seriously affect training efficiency or even cause the process to freeze. It is difficult to use for a long time in a distributed training scenario with a large-scale computing device cluster.

[0051] 2. The inconsistencies in the indicator systems of various training frameworks (such as Megatron-LM, DeepSpeed, PaLM, etc.) lead to differences in the data formats of different indicators and a lack of a unified mechanism for indicator collection, storage, and analysis.

[0052] 3. Unable to automatically generate and execute optimization strategies.

[0053] The third type is metric monitoring based on training logs. During training, training parameters, metrics, model versions, etc., are recorded in the training logs. After the training logs are reported, users manually determine the computing devices with low performance.

[0054] The disadvantage of this type of solution is:

[0055] 1. Focuses on experimental layer tracking, but does not cover the real-time behavior of cluster resources. For example, in distributed large model training, only the final loss and accuracy are recorded, which cannot identify the resource heterogeneity of each computing device or the difference in training progress.

[0056] 2. Unable to obtain underlying hardware metrics (such as GPU utilization, video memory usage, communication time, etc.) that carry timing information (indicating the training phase) and location information (indicating the computing device).

[0057] 3. Lack of the ability to automatically identify low-performance computing devices. When faced with large-scale computing device clusters, it is difficult to identify potential low-performance computing devices in a timely manner and take optimization strategies.

[0058] In view of this, this disclosure proposes a performance monitoring system and a performance monitoring method. The system is adaptable to computing device clusters of any size, can monitor the training performance indicators of computing devices, analyze whether computing devices need optimization, generate corresponding optimization strategies and execute them, thus meeting the performance monitoring, analysis and optimization needs of large-scale computing device clusters in distributed model training scenarios.

[0059] Figure 1 This illustrates an exemplary application scenario of a performance monitoring system according to embodiments of the present disclosure.

[0060] like Figure 1 As shown, the performance monitoring system includes a cluster of computing devices and a main control device.

[0061] The computing device cluster may include any number of computing devices, ranging from tens to tens of thousands. This disclosure does not limit the specific number of computing devices in the cluster. The computing devices may be chips with computing capabilities, such as CPUs or GPUs. This disclosure does not limit the specific type of computing device.

[0062] The main control device can be a terminal device or a server. This disclosure does not limit the specific type of the main control device.

[0063] The master control device can generate multiple training tasks based on the training objectives of a large model and distribute these tasks to multiple computing devices in the computing device cluster. Each computing device executes its assigned training task, while simultaneously collecting training performance metrics and transmitting them to the master control device for storage.

[0064] The main control device automatically generates and executes optimization strategies based on the stored training performance metrics.

[0065] Figure 2 A schematic diagram showing the structure of a performance monitoring system according to an embodiment of the present disclosure is provided.

[0066] like Figure 2 As shown, in one possible implementation, the system includes a cluster of computing devices for performing training tasks and a main control device, each computing device including a data acquisition server.

[0067] The data acquisition server is used to collect the training performance indicators of the computing device when it performs training tasks, and output the collected training performance indicators to the main control device.

[0068] The main control device is used to receive and store training performance indicators, analyze the stored training performance indicators to determine whether each computing device meets the preset conditions, obtain the judgment result, determine at least one computing device in the computing device cluster to be optimized based on the judgment result, generate an optimization strategy based on the computing device to be optimized and the usage of the computing device cluster, and execute the optimization strategy.

[0069] For example, each computing device may include a data acquisition server. The data acquisition server is used to collect training performance metrics of the computing device. Taking a GPU as an example, the training performance metrics that can be collected include hardware metrics such as GPU bandwidth, GPU utilization, and GPU temperature; system metrics such as memory usage, memory bandwidth, communication latency, blocking duration, network communication latency, and disk communication latency; and model metrics such as model loss, training time, and data loading time. This embodiment of the disclosure does not limit the specific type or number of training performance metrics collected by the data acquisition server.

[0070] In a computing device cluster, only some computing devices may perform training tasks. Therefore, when computing devices are performing training tasks, the data acquisition server can be used to collect training performance indicators. When computing devices are not performing training tasks, the data acquisition server does not need to work, so as to save power consumption of the computing device cluster.

[0071] After the acquisition server collects the training performance metrics, it can output these metrics to the main control device. Furthermore, when outputting the training performance metrics, it can simultaneously output the timestamp of the collected metrics and information such as the training stage. To save communication bandwidth, the training performance metrics can be compressed before being transmitted to the main control device. Compressing the training performance metrics can be implemented using existing technology, which will not be elaborated upon here.

[0072] In one example, the data acquisition server can transmit information to the master control device through communication frameworks such as gRPC and Kafka.

[0073] Figure 3 A schematic diagram showing the structure of a master control device according to an embodiment of the present disclosure is provided.

[0074] like Figure 3 As shown, the main control device may include a processing module, an analysis engine, a strategy engine, a visualization module, and a database. The processing module is a hardware chip (such as a CPU), while the analysis engine, strategy engine, visualization module, and database are software modules.

[0075] The processing module communicates with the data acquisition server to receive training performance metrics and store them in the database.

[0076] The analysis engine accesses the database to obtain training performance metrics and performs analysis to determine whether each computing device meets preset conditions. Based on the judgment result, at least one computing device in the computing device cluster that is causing a performance bottleneck is identified. The identified computing device to be optimized can be one or more computing devices executing training tasks. A computing device is considered to be causing a performance bottleneck if it meets the preset conditions. The preset conditions may include training performance metrics exceeding / falling below preset thresholds, etc. This embodiment of the disclosure does not limit the specific content of the preset conditions.

[0077] The strategy engine generates optimization strategies based on the usage of the computing device cluster and the computing devices to be optimized as determined by the analysis engine.

[0078] The processing module can improve the model training performance of the computing device cluster by executing optimization strategies to schedule computing devices.

[0079] Those skilled in the art should understand that Figure 3 The structure shown is merely an example of the main control device. In practical applications, the main control device can be implemented using more or fewer modules. The aforementioned software modules can also be replaced by hardware modules with the same function, and similarly, hardware modules can be replaced by software modules with the same function. As long as the main control device has the above-mentioned functions, this disclosure does not limit the specific implementation method of the main control device.

[0080] The performance monitoring system according to an embodiment of this disclosure includes a cluster of computing devices for performing training tasks and a main control device. Each computing device includes a data acquisition server, which is used to collect training performance indicators of the computing device when it performs a training task, and output the collected training performance indicators to the main control device. The main control device is used to receive and store the training performance indicators, analyze the stored training performance indicators to determine whether each computing device meets preset conditions, obtain a judgment result, determine at least one computing device in the computing device cluster to be optimized based on the judgment result, generate an optimization strategy based on the computing device to be optimized and the usage of the computing device cluster, and execute the optimization strategy. This method deploys an independent data acquisition server on each computing device, and aggregates the training performance data collected by each data acquisition server through the main control device, realizing real-time collection and uploading of key indicators during the training process; it releases the indicator collection pressure of the main control device to the computing devices, reduces the workload of the main control device, and is suitable for distributed model training scenarios of larger-scale computing device clusters. This system can monitor the training performance indicators of computing devices, analyze whether the computing devices need optimization, generate corresponding optimization strategies and execute them, meet the performance monitoring, analysis and optimization needs of large-scale computing device clusters in distributed model training scenarios, improve overall training efficiency, reduce resource waste and reduce manual intervention debugging costs.

[0081] In one possible implementation, the data collection server includes a unified metric abstraction layer for defining at least one metric interface common to different training frameworks.

[0082] The data collection server is specifically used to collect training performance metrics of the computing device through the metric interface. Each metric interface corresponds to the collection of one type of training performance metric.

[0083] For example, during model training, different computing devices may use different training frameworks. Commonly used training frameworks include Megatron and DeepSpeed. The data acquisition server may include a unified metric abstraction layer (Metric Abstraction Layer) to define at least one metric interface common to different training frameworks. The data acquisition server can collect training performance metrics from the computing device it is using these metric interfaces; each metric interface corresponds to the collection of one training performance metric.

[0084] In one possible implementation, the data collection server is specifically used for,

[0085] If, during the execution of a training task on the computing device, the name of any training performance metric corresponding to a certain metric interface appears, then that metric interface is used to collect data on that training performance metric once.

[0086] For example, during the training process, the computing device will continuously generate data. If the data collection server detects the name of any training performance indicator corresponding to any indicator interface during the training process of the computing device, it can use that indicator interface to collect data on that training performance indicator.

[0087] This approach decouples the data acquisition server from different training frameworks. It also ensures compatibility with various computing devices, facilitates deployment and migration across different large-scale model training environments, and avoids intrusive modifications to existing training code.

[0088] In one possible implementation, the data collection server integrates at least one metric plugin.

[0089] The data collection server is specifically used to collect training performance metrics of the computing device through metric plugins, with each metric plugin corresponding to the collection of one type of training performance metric.

[0090] For example, the metric interfaces in the unified metric abstraction layer of the acquisition server can be predefined. In practical applications, users may want the acquisition server to collect some custom training performance metrics. In this case, metric plugins can be integrated into the acquisition server. The acquisition server can collect the training performance metrics of the computing device through these plugins, with each plugin corresponding to one type of training performance metric.

[0091] In one possible implementation, the data collection server is specifically used for,

[0092] If the name of any training performance metric corresponding to a certain metric plugin appears during the execution of a training task on the computing device, then use that metric plugin to collect data on that training performance metric once.

[0093] For example, during the training process, the computing device will continuously generate data. If the data collection server detects the name of any training performance indicator corresponding to a certain indicator plugin during the training process of the computing device, it can use that indicator plugin to collect data on that training performance indicator.

[0094] During the training process on the computing device, if the user needs to collect new metrics, a new metric plugin can be integrated into the metric collection server to expand the server's metric collection capabilities in real time. This makes the metric collection on the server more flexible.

[0095] In one possible implementation, the master control device includes a time-series database.

[0096] The main control device is specifically used to add time-series information to the received training performance indicators to update the training performance indicators, store the updated training performance indicators in the time-series database, select training performance indicators within the target time period from the time-series database, analyze the selected training performance indicators, and determine at least one computing device in the computing device cluster to be optimized.

[0097] For example, Figure 3 The database shown can be a time-series database (such as Prometheus, InfluxDB, etc.). The difference between a time-series database and a regular database is that a time-series database stores not only the data itself but also its temporal information. The main control device can first add temporal information to the received training performance metrics, then store the updated training performance metrics in the time-series database. Adding the temporal information can be done by converting timestamps into easily identifiable year / month / day format time-series information for easier subsequent display.

[0098] Those skilled in the art should understand that timestamps can also be directly used as time-series information to add time-series information to the received training performance metrics, and the updated training performance metrics can be added to the time-series database. When the training performance metrics need to be displayed later, the timestamps can be converted to year / month / day format. The specific content of the time-series information is not limited in the embodiments of this disclosure.

[0099] Furthermore, the training performance metrics collected by the acquisition server may include categories that users are not interested in. These metrics can be filtered before being stored in the time-series database to prevent users from storing training performance metrics they are not interested in.

[0100] Since the training performance metrics in the time-series database are updated and carry temporal information, the master control device can select training performance metrics for a target time period from the database. The master control device can then analyze the selected training performance metrics to determine the computing equipment to be optimized. By flexibly selecting the target time period, analysis can be performed using training performance metrics collected within a single training phase, multiple training phases within a single training process, or training performance metrics collected across multiple training processes, adapting to diverse application scenarios.

[0101] Furthermore, after selecting training performance metrics for the target time period from the time-series database, the main control device can first perform a validity check to determine whether the training performance metrics are complete and not empty. If the training performance metrics are complete and not empty, the validity check passes, and the training performance metrics can be used for analysis.

[0102] The time-series database can be integrated with a real-time indicator streaming processing framework for rapid indicator access, enabling the main control device to process data at the second level and detect device anomalies. Therefore, the performance monitoring system offers better real-time performance, allowing online monitoring of the training process for timely warnings and dynamic optimization.

[0103] In one possible implementation, the data collection server is specifically used for,

[0104] Cluster the selected training performance metrics of the same type, and obtain the corresponding cluster for each training performance metric;

[0105] Calculate the threshold corresponding to each training performance metric based on the cluster corresponding to each training performance metric.

[0106] For each computing device, at least one training performance metric of the selected computing device and the corresponding threshold are used to determine whether the computing device meets the preset conditions.

[0107] When the computing device meets the preset conditions, the computing device is identified as the computing device to be optimized.

[0108] For example, more than one training performance metric may be selected for analysis. Each training performance metric may also have more than one instance. The data collection server can cluster the selected training performance metrics of the same type, generating a corresponding cluster for each training performance metric.

[0109] Figure 4 This diagram illustrates clustering of selected training performance metrics according to an embodiment of the present disclosure.

[0110] like Figure 4 As shown, assuming that computing devices 1-2 perform training tasks, the time series database stores the training performance indicators a1 and b1 collected by computing device 1 at time T1, the training performance indicators a2 and b2 collected by computing device 1 at time T2, the training performance indicators a3 and b3 collected by computing device 2 at time T1, and the training performance indicators a4 and b4 collected by computing device 2 at time T2.

[0111] Assuming the target time period includes T1, the training performance metrics involved in the analysis include a1, b1, a3, and b3. After clustering the training performance metrics of the same type, we obtain cluster 1 corresponding to the GPU utilization rate of the type, which includes training performance metrics a1 and a3, and cluster 2 corresponding to the communication latency of the type, which includes training performance metrics b1 and b3.

[0112] Based on the clusters corresponding to each training performance metric, the threshold corresponding to that training performance metric can be calculated. Figure 4In the example, threshold 1 corresponding to GPU utilization can be calculated based on cluster 1, and threshold 2 corresponding to communication latency can be calculated based on cluster 2.

[0113] In one possible implementation, the threshold for each training performance metric is equal to the mean, median, variance, or standard deviation of the training performance metrics within the corresponding cluster.

[0114] For example, the threshold can be calculated dynamically. For instance, the threshold could be equal to the mean, median, variance, or standard deviation of all metrics within a cluster.

[0115] Those skilled in the art will understand that in practical applications, the threshold can be calculated in many other ways, as long as the threshold reflects the average training performance of the computing device performing the training task. Compared to using a static threshold, the dynamic threshold calculated using the embodiments of this disclosure provides higher accuracy in identifying the computing device to be optimized. It can automatically adapt to different cluster sizes and training task characteristics, avoiding false positives and false negatives. It can accurately distinguish between "low performance caused by structural differences" and "occasional jitter," improving the accuracy and adaptability of detection, and is particularly suitable for large-scale heterogeneous cluster scenarios. Therefore, it has higher accuracy and robustness.

[0116] For each computing device, at least one training performance metric and its corresponding threshold can be used to determine whether the device meets preset conditions. If the device meets the preset conditions, it is considered a performance bottleneck for model training and can be identified as a device to be optimized.

[0117] In one possible implementation, the preset conditions include: the difference between the N training performance indicators of the computing device and the corresponding threshold is less than M times the corresponding threshold, where M is greater than 1 and N is a positive integer.

[0118] For example, one possible preset condition is that the computing device has at least N training performance metrics whose difference from the corresponding threshold is less than M times the corresponding threshold. Here, M can be set according to the application scenario requirements, and N can be flexibly adjusted based on the number of training performance metrics from the current computing device among those participating in the analysis.

[0119] by Figure 4 For example, among the training performance metrics selected for analysis, there are two metrics from computing device 1. Therefore, N can be a value less than or equal to 2.

[0120] The difference between the training performance metric and its corresponding threshold represents the deviation between the training performance of the current computing device and the average training performance of all computing devices performing the training task. A large deviation indicates poor training performance of the current computing device.

[0121] Assuming M=2 and N=1, computing device 1 satisfies the preset condition when the difference between one training performance metric and its corresponding threshold is less than twice the threshold. For example, if the difference between training performance metric a1 and threshold 1 is less than twice the threshold 1, or the difference between training performance metric b1 and threshold 2 is less than twice the threshold 2, computing device 1 satisfies the preset condition.

[0122] Furthermore, more stringent preset conditions can be set. For example, when the computing device has K training performance indicators, the preset conditions may also include: N / K is greater than a preset first value. The first value may be, for example, 50%. This embodiment of the present disclosure does not limit the way the first value is set. That is, assuming K=2, N=1, and the computing device has two training performance indicators a1 and b1, training performance indicator a1 corresponds to threshold 1, and training performance indicator b1 corresponds to threshold 2, then the computing device 1 satisfies the preset conditions only when the difference between training performance indicator a1 and threshold 1 is less than twice the threshold 1, and the difference between training performance indicator b1 and threshold 2 is less than twice the threshold 2.

[0123] In practical applications, preset conditions can be flexibly modified according to user expectations. More lenient preset conditions are used when users expect higher training efficiency; more stringent preset conditions are used when users expect more accurate judgment of the computing device to be optimized. This disclosure does not limit the specific preset conditions used when judging the computing device to be optimized.

[0124] Those skilled in the art will understand that a static threshold can also be used to determine the computing device to be optimized. If the value of the training performance indicator significantly exceeds the corresponding static threshold, for example, if the difference between the static threshold and the value is greater than a preset second value, then the computing device is determined to be the computing device to be optimized. This disclosure does not limit the specific method for determining the threshold.

[0125] When identifying computing devices to be optimized, issues such as low GPU utilization or high communication latency can be recorded. If it's necessary to display the optimized computing devices, these issues can be shown simultaneously. A view can then be used to compare the issues of different computing devices. This approach offers greater interpretability, helps users understand the specific reasons for the optimization, and reduces manual troubleshooting and debugging time.

[0126] In one possible implementation,

[0127] When there are idle computing devices in the computing device cluster, optimization strategies include using the idle computing devices to perform training tasks instead of the computing devices to be optimized.

[0128] When there are no idle computing devices in the computing device cluster, optimization strategies include restarting the computing device to be optimized.

[0129] For example, the content of the optimization strategy generated by the main control device is shown below:

[0130] When there are idle computing devices in the computing device cluster, optimization strategies may include using the idle computing devices to perform training tasks in place of the computing device to be optimized. When there are no idle computing devices in the computing device cluster, optimization strategies may include restarting the computing device to be optimized.

[0131] Those skilled in the art should understand that when there are idle computing devices in the computing device cluster, the optimization strategy may also include restarting the computing device to be optimized. The embodiments of this disclosure do not limit the specific content of the optimization strategy generated by the main control cabinet device.

[0132] In one possible implementation, the master control device is also used to display to the user at least one of the following through a visualization module: at least one computing device to be optimized, training performance metrics received by the master control device, and optimization strategies executed.

[0133] For example, the main control device can use a visualization module to display to the user the computing devices to be optimized, training performance metrics, and the optimization strategies implemented. The visualization module functions similarly to a user interface (UI), allowing users to query training performance metrics collected within any time period, from any computing device, and at any training stage. It also supports tracing training performance metrics collected during different training processes and displaying them in a comparative format.

[0134] Furthermore, the visualization module can also display overall training efficiency, etc. This embodiment of the disclosure does not limit the specific content displayed by the visualization module, as long as it helps the user to confirm the model training status.

[0135] This disclosure also proposes a performance monitoring method. Figure 5 A schematic diagram illustrating the flow of a performance monitoring method according to an embodiment of the present disclosure is shown.

[0136] like Figure 5 As shown, in one possible implementation, the method is applied to a performance monitoring system, the system including a cluster of computing devices for performing training tasks and a main control device, each computing device including a data acquisition server, and the method including:

[0137] Step S51: When the server is performing a training task on the computing device, it collects the training performance indicators of the computing device and outputs the collected training performance indicators to the main control device.

[0138] Step S52: The main control device receives and stores training performance indicators, analyzes the stored training performance indicators to determine whether each computing device meets the preset conditions, obtains the judgment result, and determines at least one computing device in the computing device cluster to be optimized based on the judgment result; generates an optimization strategy based on the computing device to be optimized and the usage of the computing device cluster; and executes the optimization strategy.

[0139] In one possible implementation, the data acquisition server includes a unified metric abstraction layer for defining at least one metric interface common to different training frameworks; the data acquisition of training performance metrics of the computing device includes: acquiring the training performance metrics of the computing device through the metric interface, with each metric interface corresponding to the acquisition of one training performance metric.

[0140] In one possible implementation, collecting the training performance metrics of the computing device through the metric interface includes: when the name of any training performance metric corresponding to any metric interface appears during the execution of the training task on the computing device, using that metric interface to collect the training performance metric once.

[0141] In one possible implementation, the data acquisition server integrates at least one indicator plugin, and the data acquisition of the training performance indicators of the computing device includes: acquiring the training performance indicators of the computing device through the indicator plugin, with each indicator plugin corresponding to the acquisition of one training performance indicator.

[0142] In one possible implementation, the step of collecting training performance metrics of the computing device through the metric plugin includes: when the name of a training performance metric corresponding to any metric plugin appears during the execution of a training task on the computing device, the metric plugin is used to collect the training performance metric once.

[0143] In one possible implementation, the master control device includes a time-series database. Receiving and storing training performance indicators includes: adding time-series information to the received training performance indicators to obtain updated training performance indicators, and storing the updated training performance indicators in the time-series database. Analyzing the stored training performance indicators to determine at least one computing device to be optimized in the computing device cluster includes: selecting training performance indicators within a target time period from the time-series database, analyzing the selected training performance indicators, and determining at least one computing device to be optimized in the computing device cluster.

[0144] In one possible implementation, the step of analyzing the selected training performance metrics to determine at least one computing device to be optimized in the computing device cluster includes: clustering the selected training performance metrics of the same type to obtain a corresponding cluster for each training performance metric; calculating a threshold corresponding to each training performance metric based on the cluster corresponding to each training performance metric; for each computing device, using at least one selected training performance metric of the computing device and the corresponding threshold, determining whether the computing device meets a preset condition; and when the computing device meets the preset condition, determining the computing device as the computing device to be optimized.

[0145] In one possible implementation, the preset conditions include: the difference between the N training performance indicators of the computing device and the corresponding threshold is less than M times the corresponding threshold, where M is greater than 1 and N is a positive integer.

[0146] In one possible implementation, the threshold corresponding to each training performance metric is equal to the mean, mode, median, variance, or standard deviation of the training performance metrics within the corresponding cluster.

[0147] In one possible implementation, when there are idle computing devices in the computing device cluster, the optimization strategy includes using the idle computing devices to perform the training task in place of the computing device to be optimized; when there are no idle computing devices in the computing device cluster, the optimization strategy includes restarting the computing device to be optimized.

[0148] In one possible implementation, the method further includes: the master control device displaying to the user at least one of the at least one computing device to be optimized, the training performance indicators received by the master control device, and the optimization strategy executed, through a visualization module.

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0150] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A performance monitoring system, characterized in that, The system includes a cluster of computing devices for performing training tasks and a main control device. Each computing device includes a data acquisition server. The data acquisition server is used to collect training performance indicators of the computing device when the computing device performs a training task, and output the collected training performance indicators to the main control device. The main control device is used to receive and store training performance indicators, analyze the stored training performance indicators to determine whether each computing device meets preset conditions, obtain a judgment result, and determine at least one computing device to be optimized in the computing device cluster based on the judgment result. An optimization strategy is generated based on the usage of the computing device to be optimized and the computing device cluster. Implement the optimization strategy.

2. The system according to claim 1, characterized in that, The data collection server includes a unified indicator abstraction layer, which is used to define at least one indicator interface common to different training frameworks. The data collection server is specifically used to collect training performance metrics of the computing device through the metric interface, with each metric interface corresponding to the collection of one type of training performance metric.

3. The system according to claim 2, characterized in that, The data collection server is specifically used for, If, during the execution of a training task on the computing device, the name of any training performance metric corresponding to a certain metric interface appears, then that metric interface is used to collect data on that training performance metric once.

4. The system according to claim 2, characterized in that, The data collection server integrates at least one indicator plugin. The data collection server is specifically used to collect training performance metrics of the computing device through the metric plugins, with each metric plugin corresponding to the collection of one type of training performance metric.

5. The system according to claim 4, characterized in that, The data collection server is specifically used for, If the name of any training performance metric corresponding to a certain metric plugin appears during the execution of a training task on the computing device, then use that metric plugin to collect data on that training performance metric once.

6. The system according to claim 1, characterized in that, The main control device includes a time-series database. The main control device is specifically used to add time series information to the received training performance indicators to obtain updated training performance indicators, and to store the updated training performance indicators into the time series database. The training performance metrics within the target time period are selected from the time series database, and the selected training performance metrics are analyzed to determine at least one computing device in the computing device cluster that needs to be optimized.

7. The system according to claim 6, characterized in that, The data collection server is specifically used for, Cluster the selected training performance metrics of the same type, and obtain the corresponding cluster for each training performance metric; Calculate the threshold corresponding to each training performance metric based on the cluster corresponding to each training performance metric. For each computing device, at least one training performance metric of the selected computing device and the corresponding threshold are used to determine whether the computing device meets the preset conditions. When the computing device meets the preset conditions, the computing device is identified as the computing device to be optimized.

8. The system according to claim 7, characterized in that, The preset conditions include: the difference between the N training performance indicators of the computing device and the corresponding threshold is less than M times the corresponding threshold, where M is greater than 1 and N is a positive integer.

9. The system according to claim 7, characterized in that, The threshold for each training performance metric is equal to the mean, mode, median, variance, or standard deviation of the training performance metrics within the corresponding cluster.

10. The system according to claim 1, characterized in that, When there are idle computing devices in the computing device cluster, the optimization strategy includes using the idle computing devices to perform training tasks instead of the computing devices to be optimized; When there are no idle computing devices in the computing device cluster, the optimization strategy includes restarting the computing device to be optimized.

11. The system according to claim 1, characterized in that, The main control device is also used to display to the user at least one of the at least one computing device to be optimized, the training performance indicators received by the main control device, and the optimization strategy executed, through a visualization module.

12. A performance monitoring method, characterized in that, The method is applied to a performance monitoring system, the system including a cluster of computing devices for performing training tasks and a main control device, each computing device including a data acquisition server, and the method including: When the data acquisition server performs a training task on the computing device, it collects the training performance indicators of the computing device and outputs the collected training performance indicators to the main control device. The master control device receives and stores training performance indicators, analyzes the stored training performance indicators to determine whether each computing device meets preset conditions, obtains a judgment result, and determines at least one computing device in the computing device cluster to be optimized based on the judgment result; generates an optimization strategy according to the usage of the computing device to be optimized and the computing device cluster; and executes the optimization strategy.

Images