An efficiency monitoring system of a large model training cluster

Abstract

The application discloses a kind of efficiency monitoring systems of large model training cluster, including training data acquisition module, backend processing module and client;Training data acquisition module includes monitoring hook sub-module and database;Monitoring hook sub-module is realized by rewriting the module finder of Python, the import request of the core module of the training task is intercepted, and monitoring hook is installed into target training process;Monitoring hook real-time acquisition corresponding data and store in database;Backend processing module accesses the various monitoring data collected by data acquisition module through data acquisition module interface access layer, and calculates various training efficiency related composite indexes based on preprocessed monitoring data through high-order index calculation engine.The application can be real-time monitored, non-invasive deployment and can guarantee data integrity.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and distributed computing, and in particular to a performance monitoring system for a large model training cluster. Background Art

[0002] As deep learning models continue to scale, the number of parameters in a single model has grown from millions to hundreds of billions or even trillions. Training these large-scale models typically requires hundreds of GPU servers working together, and training cycles can last from weeks to months. In such a complex distributed training environment, real-time monitoring of training performance, timely detection of anomalies, and optimization of resource utilization have become key factors influencing training success and economic benefits.

[0003] Currently, performance monitoring of large model training mainly relies on the following technical solutions:

[0004] 1. Log file-based monitoring solution

[0005] Traditional monitoring methods mainly obtain training indicators by parsing log files generated during training. The specific approach is:

[0006] (1) The training program outputs key information to the log file during operation;

[0007] (2) An independent monitoring program regularly reads and parses these log files;

[0008] (3) Extracting indicators such as the number of training iterations, loss value, and throughput through regular expressions;

[0009] (4) Storing the extracted data in a database for subsequent analysis;

[0010] This approach is characterized by its simplicity of implementation and low intrusion into the training process, but it has obvious limitations:

[0011] (1) Data delay problem: Log file writing usually has a buffering mechanism, and the monitoring program cannot obtain the latest training status in real time.

[0012] (2) Data integrity issue: When the training program terminates abnormally, the logs in the buffer may be lost, resulting in incomplete monitoring data.

[0013] (3) Parsing complexity: The log formats of different training frameworks vary greatly, and specialized parsing rules need to be developed for each framework.

[0014] (4) Resource overhead: Frequent file I / O operations increase system load, which is particularly noticeable in large-scale clusters.

[0015] 2. Monitoring solutions based on system indicators

[0016] Another common type of monitoring solution focuses on collecting system-level resource usage metrics, including:

[0017] (1) System indicators such as CPU usage, memory usage, and disk I / O rate.

[0018] (2) GPU-related indicators such as GPU utilization, video memory usage, and GPU temperature.

[0019] (3) Network indicators such as network bandwidth usage and network latency.

[0020] This type of solution typically uses mature monitoring tools such as Prometheus and Grafana, which can provide rich visualization interfaces and alarm functions. However, its limitations are:

[0021] (1) Lack of training semantics: System indicators cannot reflect the actual progress and effects of training, such as the inability to obtain detailed information on training events such as storage record points.

[0022] (2) Coarse-grained metrics: It is impossible to obtain fine-grained performance data at the training iteration level, such as the start and end time of each iteration and the throughput.

[0023] (3) Poor correlation: It is difficult to establish a direct correlation between system indicators and training effects.

[0024] 3. Solutions based on built-in monitoring in the training framework

[0025] Some deep learning frameworks provide built-in monitoring capabilities, such as TensorBoard and Weights & Biases. The characteristics of these solutions are:

[0026] Able to record detailed indicators during training, including loss curve, gradient distribution, model parameter changes, etc.

[0027] Provides rich visualization functions to facilitate researchers to analyze the training process;

[0028] It is tightly integrated with the training code and can obtain detailed information inside the training framework.

[0029] However, this approach also has significant drawbacks:

[0030] (1) Highly invasive code: Monitoring logic needs to be explicitly added to the training code;

[0031] (2) Framework dependency: usually only applicable to specific training frameworks;

[0032] (3) High performance overhead: Detailed monitoring information recording will significantly affect training performance;

[0033] (4) Data storage is scattered: Monitoring data from different experiments are stored in different locations, making unified management difficult;

[0034] Therefore, existing performance monitoring of large-scale model training has a fundamental limitation: it is impossible to obtain key indicators of the training process in real time, accurately, and completely without modifying the training code.

[0035] Specifically:

[0036] (1) Poor timeliness: Solutions based on log parsing have inherent delay issues.

[0037] (2) Low accuracy: System-level monitoring cannot reflect the true status of training.

[0038] (3) Poor integrity: Existing solutions have difficulty in ensuring data integrity during training anomalies.

[0039] (4) Highly invasive: requires modifying the training code or relying on the monitoring function of a specific framework.

[0040] (5) Poor scalability: It is difficult to adapt to different training frameworks and deployment environments.

[0041] These limitations are particularly prominent in large-scale distributed training environments, seriously affecting the optimization of training efficiency and the rapid location of problems. Summary of the Invention

[0042] To address the shortcomings of the existing technology, the present invention proposes a performance monitoring system for a large model training cluster. The specific technical solution is as follows:

[0043] A performance monitoring system for a large model training cluster, including a training data acquisition module, a backend processing module, and a client;

[0044] The training data acquisition module includes a monitoring hook submodule and a database; the monitoring hook submodule intercepts the import request of the core module of the training task by rewriting Python's module finder and installs the monitoring hook into the target training process; the monitoring hook can collect corresponding data in real time and store it in the database;

[0045] The backend processing module includes a data acquisition module interface access layer and a high-level indicator calculation engine. The data acquisition module interface access layer obtains various monitoring data collected by the data acquisition module; the high-level indicator calculation engine calculates various composite indicators related to training effectiveness based on the pre-processed monitoring data;

[0046] The client is used to display basic training parameters, a training lifecycle pipeline diagram generated based on basic event data, and high-level training performance indicators.

[0047] Furthermore, the monitoring hooks include a checkpoint operation monitoring hook, an error event monitoring hook, a lifecycle hook, and an iteration output monitoring hook;

[0048] The checkpoint operation monitoring hook is used to monitor the save and load process of the checkpoint and save the data according to the checkpoint log data structure;

[0049] The error event monitoring hook is used to automatically capture and record errors and save the data as a standard error log data structure;

[0050] The lifecycle hook includes a job start hook and a job end hook. The job start hook is used to collect and record basic job information when the training task is started, and the job end hook is used to record the task end time;

[0051] The iteration output monitoring hook is used to collect training iteration data and save the collected data into a standard iteration data structure.

[0052] Furthermore, the checkpoint operation monitoring hook monitors the checkpoint operation by hijacking the checkpoint module in the large model training framework;

[0053] The error event monitoring hook hijacks the Python system's exception handling function and replaces it with a new exception handling function that can automatically capture and record errors;

[0054] The iterative output monitoring hook collects training iteration data by hijacking the key logging function in the large model training framework.

[0055] Furthermore, the training iteration data collected by the iteration output monitoring hook includes the number of iterations, the time taken for a single iteration, the total number of iterations, the computational throughput, the loss value, and the learning rate.

[0056] Furthermore, the function hijacking process of the monitoring hook is as follows:

[0057] First, save the reference to the original function and define a wrapper function to replace the original implementation;

[0058] Then, the wrapper function is executed to extract training parameters and status information, calculate performance indicators, and record the data into the performance monitoring system;

[0059] Finally, call the original function to ensure that the training logic is executed normally.

[0060] Furthermore, the performance indicators include the start time of a single iteration, precise time consumption measurement, and theoretical throughput calculation based on model parameters and batch size.

[0061] Furthermore, the data acquisition module interface access layer is a unified interface for distributed queries, which can realize single-node query and multi-node data merging functions; the multi-node data merging specifically includes: simultaneously obtaining data from the databases of multiple training nodes, and deduplicating, sorting and merging the query results to ensure the consistency and integrity of the final data.

[0062] Furthermore, when the backend processing module obtains monitoring data from the datasets of multiple training nodes, it adopts an incremental data collection strategy, that is, by maintaining the last updated timestamp of each job, it ensures that only new data since the last collection is obtained.

[0063] Furthermore, when the back-end processing module collects data, it performs coordinated collection of multiple types of data to ensure that different types of data can be updated synchronously, and adopts differentiated collection strategies based on the update frequency characteristics of different data types.

[0064] Furthermore, the composite indicators related to training effectiveness include basic performance indicators and cluster effectiveness indicators;

[0065] The basic performance indicators include three indicators that reflect the training speed: average iteration time, token processing throughput, and average GPU utilization:

[0066] The calculation formula of the cluster efficiency index is as follows:

[0067] MFU = E1 × E2 × E3

[0068] E1 = non-fault time / total time

[0069] E2 = Number of floating-point operations for a complete training session / (GPU nominal power × number of GPUs × total effective training time)

[0070] E3 = Effective training time / non-failure time

[0071] Among them, MFU represents the cluster performance indicator, E1 represents the device online availability rate, E2 represents the single-card software stack utilization rate, and E3 represents the model training parallel efficiency.

[0072] The beneficial effects of the present invention are as follows:

[0073] 1. Zero-latency real-time monitoring: By directly collecting data from within the training framework, this method can obtain relevant indicators at the moment of the start or completion of events such as training iterations and record point saves, achieving true real-time monitoring and avoiding the file I / O delays of traditional log parsing solutions. Compared with traditional log parsing solutions, monitoring latency is reduced from seconds to milliseconds.

[0074] 2. Guaranteed Data Integrity: This invention uses function-level interception to ensure that every critical operation is accurately recorded and the data is fully consistent with the framework's internal data. Even in the event of abnormal training termination, the data integrity of completed operations is guaranteed. Compared to log parsing solutions (which can lose data for that iteration if the logging module has not yet executed), this invention prevents data loss.

[0075] 3. Non-intrusive deployment: By using dynamic injection technology, this invention only requires pre-deployment through mirroring, without modifying the user's training code. It can monitor any existing training code without redevelopment or recompilation, thus significantly reducing deployment costs. Compared with internal monitoring solutions such as Tensorboard, users do not need to pay the learning cost of writing log output code, simplifying user operations.

[0076] 4. Support for Multiple Frameworks: This invention utilizes a universal interception mechanism designed based on the characteristics of the Python language to support mainstream training frameworks such as PyTorch, TensorFlow, and Megatron. Expanding to new frameworks simply requires developing appropriate monitoring hook functions. Compared to log collection methods, this significantly reduces development and maintenance costs.

[0077] 5. Centralized data storage and management: This invention uses the backend to directly store collected data in a database, supporting the centralized storage of large-scale training data for the platform. Compared to internal monitoring solutions such as Tensorboard, where log files are distributed across the user's own storage, this solution can centrally collect training data from all users, greatly facilitating the monitoring and analysis of cluster-level performance indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 This is an architectural diagram of the performance monitoring system for a large model training cluster of the present invention.

[0079] Figure 2 This is the architecture diagram of the training data acquisition module.

[0080] Figure 3 Schematic diagram of the data collection process for the training data collection module.

[0081] Figure 4 A schematic diagram of the backend processing module.

[0082] Figure 5 Schematic diagram of the injection and monitoring hook installation process of the performance monitoring system of the large model training cluster of the present invention. DETAILED DESCRIPTION

[0083] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more apparent. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0084] like Figure 1 As shown, the performance monitoring system of the large model training cluster of the present invention includes a training data acquisition module, a back-end processing module and a client.

[0085] 1. Training Data Collection Module

[0086] like Figure 2 As shown, the training data acquisition module includes a monitoring hook submodule and a database.

[0087] The training data collection module uses an HTTP REST API to ensure reliable communication with the backend processing module. The sending process includes timeout control and error handling, so that even in the event of network anomalies, the normal start of the training process will not be blocked.

[0088] 1. Monitoring hook submodule

[0089] The monitoring hook submodule rewrites Python's module finder to intercept import requests for core training modules and install monitoring hooks into the target training process. These monitoring hooks collect data in real time and store it in a database. These monitoring hooks include checkpoint monitoring hooks, error event monitoring hooks, iteration output monitoring hooks, and lifecycle hooks.

[0090] (1) Checkpoint operation monitoring hook

[0091] The checkpoint operation monitoring hook is used to monitor the checkpoint saving and loading process, that is, to collect data specifically for the saving and loading process of the model state, including operation type, file path, start time, end time, file size, corresponding number of iterations, and other information, and save it according to the checkpoint log data structure.

[0092] The monitoring of checkpoint operations is specifically achieved by hijacking the checkpointing module in the Megatron framework. After the module is imported, the checkpoint operation monitoring hook automatically locates key functions such as save_checkpoint and load_checkpoint that are responsible for the persistence of the model state.

[0093] Checkpoint save monitoring is implemented by recording the start time before a function call and the end time and file size after the function completes. This precise time measurement allows us to calculate the exact duration of a checkpoint save, which is crucial for analyzing training efficiency.

[0094] Checkpoint loading operation monitoring is similar, focusing on loading time-consuming and file integrity verification. For distributed training environment, the status information of checkpoint synchronization between different nodes is also recorded.

[0095] Checkpoint operation monitoring hook is set with exception handling mechanism, ensuring that even in the case of checkpoint operation failure, the relevant error information can be recorded correctly. This includes failure type, error time, error reason, etc., providing detailed diagnostic information for fault analysis.

[0096] Monitoring data is stored in a standardized data model, supporting time series query of checkpoint operation history records, facilitating analysis of checkpoint strategy effect and optimization of storage system performance.

[0097] (2) Error event monitoring hook

[0098] Error event monitoring hook is achieved by replacing Python system exception handling function to automatically capture and record errors, saved as standard error log data structure, including timestamp, error type, error message, stack trace, iteration number at the time of occurrence, process ranking and other detailed information.

[0099] When any uncaught exception occurs in the training process, the error event monitoring hook is automatically called, and the original exception handling logic is called after recording the error information.

[0100] Error information collection includes accurate identification of exception type, complete recording of error message, detailed tracking of call stack, etc. By obtaining the training context information at the time of error occurrence, such as iteration number, process ranking, etc., it is extremely important for fault location in distributed training environment.

[0101] The detailed tracking of stack is recorded in the standard Python traceback format, which retains the complete function call chain information, facilitating developers to diagnose problems and debug code.

[0102] In order to ensure the stability of the monitoring system itself, the error recording process adopts multiple exception protection mechanism. Even if an error occurs in the monitoring system, it will not affect the original exception handling process, ensuring that the error information of the training process can be normally output. Through the timestamp recorded, the error event data and the training iteration data are time-related, supporting the analysis of the relationship between error occurrence mode and training stage, providing data support for training strategy optimization.

[0103] (3) Iteration output monitoring hook

[0104] The iteration output monitoring hook collects training iteration data by hijacking the key logging functions in the Megatron training framework, and saves the collected data as a standard iteration data structure. The collected data includes key indicators such as the number of iterations, the time taken for a single iteration, the total number of iterations, the calculation throughput, the loss value, and the learning rate.

[0105] The function hijacking mechanism is automatically triggered after the Megatron module is loaded. In this embodiment, the iteration output monitoring hook is located in the training_log function in the training module to collect training iteration data. This is the core function of the Megatron framework for recording training progress.

[0106] The monitoring hook hijacking process first saves a reference to the original function and then defines a wrapper function to replace the original implementation. When executed, the wrapper function extracts training parameters and status information, calculates performance metrics, logs the data to the monitoring system, and finally calls the original function to ensure the normal execution of the training logic.

[0107] Performance metric calculations include the start time of a single iteration, precise time measurement, and theoretical throughput calculations based on model parameters and batch size. These calculations are performed within the context of function calls, enabling the most accurate real-time data.

[0108] (4) Lifecycle hooks

[0109] Lifecycle hooks, including job start hook and job end hook.

[0110] The job start hook is responsible for collecting and recording basic job information when a training task starts (that is, before the training framework is fully initialized). This hook analyzes environment variables to obtain necessary configuration information and proactively sends a job start event to the backend processing module. The execution timing of the job start hook is carefully designed to ensure that event recording is completed before the training framework is fully initialized, establishing a starting point for monitoring the entire training process.

[0111] The job start hook first identifies the node role, ensuring that event recording is performed only on the primary node (GROUP_RANK=0 and LOCAL_RANK=0), avoiding duplicate recording in a distributed environment. This design ensures event uniqueness while reducing network communication overhead.

[0112] Basic job information includes the job identifier, unique timestamp, start time, distributed configuration parameters (world size, number of nodes, number of GPUs per node), training framework type, model type and scale, etc. This information provides an important metadata foundation for subsequent data association and analysis.

[0113] The job end hook, by registering the end monitoring function in the native atexist function of Python, can collect the exact time point of the job end and the job task number JOB_ID to obtain the precise end time of each job. The specific process of data collection for each hook is as follows Figure 3 shown.

[0114] 2 Database

[0115] The database contains several data tables, each of which stores a specific type of data (e.g., an iteration output data table, a record point data table, etc.). Each data entry in the table is timestamped and contains the data columns from which it was collected. The database provides a unified table query interface for each data table.

[0116] 2. Backend processing module

[0117] like Figure 4 As shown, the backend processing module includes a data acquisition module interface access layer and a high-level indicator calculation engine, and obtains various monitoring data collected by the data acquisition module through the data acquisition module interface access layer.

[0118] 1. Data acquisition module interface access layer

[0119] The data acquisition module's interface access layer provides a unified interface for distributed queries, enabling both single-node query functionality and multi-node data merging. Multi-node data merging involves simultaneously acquiring data from the databases of multiple training nodes and performing deduplication, sorting, and merging operations on the query results to ensure the consistency and integrity of the final data.

[0120] The backend processing module adopts an asynchronous programming model, supports concurrent queries on multiple training nodes, and significantly improves the efficiency of data acquisition.

[0121] The backend processing module maintains the node list and connection pool internally to ensure stable communication with each training node.

[0122] The single-node query function sends SQL query requests to the specified training data collection module through the HTTP interface, supporting standard SELECT statement syntax. The query results are returned in JSON format, and the client automatically converts them into a structured data table format.

[0123] The backend processing module also provides the client with a query interface for pre-processed data for different types of monitoring data, including training iteration data acquisition, checkpoint event query, system resource indicator acquisition, etc. Each method is optimized for specific data characteristics.

[0124] The back-end processing module can ensure that in the event of network anomalies or node failures, the problematic nodes are automatically skipped and the data of available nodes are used to continue providing services, ensuring the high availability of the monitoring system.

[0125] The backend processing module also implements an intelligent polling data update mechanism, regularly obtaining the latest monitoring data from the training data acquisition module of each training node. The polling mechanism uses a configurable time interval and is dynamically adjusted based on the urgency of the training task and the frequency of data updates.

[0126] The data collection service of the back-end processing module runs in an independent asynchronous task to ensure that the main business processing flow is not blocked.

[0127] The backend processing module uses an incremental data collection strategy to retrieve monitoring data from the databases of multiple training nodes. This strategy ensures that only new data since the last collection is obtained by maintaining the last update timestamp for each job. This design significantly reduces network transmission overhead and data processing burden, making it particularly suitable for long-running, large-scale training tasks.

[0128] When the back-end processing module collects data, it also performs coordinated collection of multiple types of data to ensure that different types of data, such as training iteration data, checkpoint events, and system resource indicators, can be updated synchronously; and adopts differentiated collection strategies based on the update frequency characteristics of different data types.

[0129] The back-end processing module is equipped with fault tolerance and recovery mechanisms to ensure that data collection services can continue to operate even in the event of a single node failure or network interruption; failed collection operations are automatically recorded and compensatory data collection is performed after conditions are restored.

[0130] 2. Advanced indicator calculation engine

[0131] The high-level metrics calculation engine is responsible for calculating various composite metrics related to training performance based on preprocessed monitoring data. Using a streaming computing architecture, the calculation engine can process large amounts of raw data in real time and generate meaningful analytical results.

[0132] Composite metrics related to training performance cover multiple dimensions of performance evaluation. Basic performance indicators include average iteration time, token processing throughput, average GPU utilization, and other metrics that directly reflect training speed. These metrics provide a quantitative basis for basic evaluation of training performance.

[0133] The calculation of the cluster performance indicator (MFU) is a core component of high-level metrics. By analyzing basic indicators such as effective training time, overall training time, theoretical computing power, and fault duration, three key indicators are calculated: device online availability (E1), single-card software stack utilization (E2), and model training parallel efficiency (E3). These indicators accurately reflect the efficiency of hardware resource utilization in model training.

[0134] The calculation method of each indicator is as follows:

[0135] E1 = non-fault time / total time

[0136] The fault duration can be calculated based on the fault occurrence time and the project restart time.

[0137] E2 = Number of floating-point operations for a complete training session / (GPU nominal power × number of GPUs × total effective training time)

[0138] The total effective training time can be obtained by accumulating the time consumed by each iteration.

[0139] E3 = Effective training time / non-failure time

[0140] MFU = E1 × E2 × E3

[0141] Through the detailed implementation methods described above, the present invention enables comprehensive, real-time, and precise monitoring of large-scale model training processes, providing strong technical support for optimizing training performance and diagnosing problems. The present invention's innovative dynamic injection technology and function hijacking mechanism achieve zero-latency real-time data collection while ensuring monitoring accuracy, significantly improving the management efficiency and operational quality of large-scale model training clusters.

[0142] 3. Client

[0143] The client is used to display basic training parameters, training lifecycle pipeline diagrams generated based on basic event data, and high-level training performance indicators.

[0144] The method for performing performance monitoring using the performance monitoring system of the large model training cluster of the present invention specifically comprises the following steps:

[0145] Step 1: Build a training image that includes a cluster performance monitoring system

[0146] 1. Choose a standard deep learning base image

[0147] Building a model training environment image requires expanding upon a standard deep learning base image. First, select an official base image that includes PyTorch and CUDA support, then install the necessary system dependencies, including the compilation toolchain and version control tools.

[0148] In this example, the performance monitoring system for a large model training cluster is developed using the Rust language, so the Rust compiler environment must be installed in the image. Use the Rustup tool to install the latest stable version of the Rust compiler and configure the corresponding environment variables to ensure that the Rust compiler can be used normally after the container is started.

[0149] 2. Install the large model training framework

[0150] In this example, we installed the Megatron training framework. We cloned the Megatron-LM source code from the official repository and installed it using Python package management tools. As the core framework for large-scale model training, Megatron's installation requires specific dependencies, including compatible PyTorch and CUDA versions.

[0151] 3. Install a performance monitoring system for large model training clusters

[0152] Obtain the project code for the performance monitoring system for a large model training cluster from the source code repository, compile the dynamic library using the Rust compiler, and install the Python bindings using the Python package manager. During the installation process, ensure that the libprobing.so dynamic library can be correctly loaded by Python.

[0153] 4. According to different deployment scenarios, configure the corresponding environment variables to activate the monitoring function of the performance monitoring system of the large model training cluster

[0154] In a Kubernetes cluster deployment environment, monitoring behavior is controlled by configuring environment variables in Job or Deployment resources.

[0155] The environment variable here is the core activation switch. Setting it to "1" enables single-process monitoring mode, and setting it to "2" enables nested process monitoring mode. The PROBING_PORT environment variable specifies the communication port for the monitoring service. The performance monitoring system automatically assigns different port numbers based on the LOCAL_RANK to avoid conflicts.

[0156] Environment variables related to distributed training include WORLD_SIZE (total number of processes), GROUP_RANK (node ​​group ranking), LOCAL_RANK (local process ranking), etc. These variables are not only used to coordinate the training process, but also for data organization and identification of the monitoring system.

[0157] Job identification-related environment variables are used to distinguish different training tasks, including JOB_ID, JOB_NAME, etc. These identifiers will serve as important indexes for data query and analysis.

[0158] Step 2: Start training and begin monitoring

[0159] 1. System injection and hook installation

[0160] First, the target training process's operating environment is detected, including its process ID, architecture, and memory layout. Using dynamic injection technology supported by the Linux system, the system source code is implanted into the target process. The performance monitoring system for the large-model training cluster is then loaded into the target training process. The performance monitoring system for the large-model training cluster automatically installs monitoring hooks, which hijack key functions in the training framework (such as those involving training iterations, GPU operations, and file I / O). These hooks ensure that the original functionality of the hijacked functions remains unchanged while recording key training data.

[0161] 2. Data Collection Execution

[0162] The monitoring hook in the monitoring hook submodule starts recording key training data, and the collected data enters the distributed database.

[0163] 3. Data processing and indicator calculation

[0164] After completing data collection, the back-end processing module collects, removes duplicates, aggregates, and other operations on the original data, and then calculates the core efficiency indicators. The specific execution process of step 2 is as follows Figure 5 shown.

[0165] Step 3: Data display

[0166] After high-level data calculations are completed, visualization is performed. Various indicators are displayed in real time through the client's web interface, providing alarm and anomaly detection functions.

[0167] Those skilled in the art will understand that the foregoing descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art will still be able to modify the technical solutions described in the foregoing examples or substitute equivalents for some of the technical features therein. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the invention shall be included within the scope of protection of the invention.

Claims

1. A performance monitoring system for a large model training cluster, characterized in that: Includes training data acquisition module, backend processing module and client; The training data acquisition module includes a monitoring hook submodule and a database; the monitoring hook submodule intercepts the import request of the core module of the training task by rewriting Python's module finder and installs the monitoring hook into the target training process; the monitoring hook can collect corresponding data in real time and store it in the database; The backend processing module includes a data acquisition module interface access layer and a high-level indicator calculation engine. The data acquisition module interface access layer obtains various monitoring data collected by the data acquisition module; the high-level indicator calculation engine calculates various composite indicators related to training effectiveness based on the pre-processed monitoring data; The client is used to display basic training parameters, a training lifecycle pipeline diagram generated based on basic event data, and high-level training performance indicators.

2. The performance monitoring system for a large model training cluster according to claim 1, characterized in that: The monitoring hooks include a checkpoint operation monitoring hook, an error event monitoring hook, a lifecycle hook, and an iteration output monitoring hook; The checkpoint operation monitoring hook is used to monitor the save and load process of the checkpoint and save the data according to the checkpoint log data structure; The error event monitoring hook is used to automatically capture and record errors and save the data as a standard error log data structure; The lifecycle hook includes a job start hook and a job end hook. The job start hook is used to collect and record basic job information when the training task is started, and the job end hook is used to record the task end time; The iteration output monitoring hook is used to collect training iteration data and save the collected data into a standard iteration data structure.

3. The performance monitoring system for a large model training cluster according to claim 2, characterized in that: The checkpoint operation monitoring hook monitors the checkpoint operation by hijacking the checkpoint module in the large model training framework; The error event monitoring hook hijacks the Python system's exception handling function and replaces it with a new exception handling function that can automatically capture and record errors; The iterative output monitoring hook collects training iteration data by hijacking the key logging function in the large model training framework.

4. The performance monitoring system for a large model training cluster according to claim 2, characterized in that: The training iteration data collected by the iteration output monitoring hook includes the number of iterations, single iteration time, total number of iterations, computational throughput, loss value, and learning rate.

5. The performance monitoring system for a large model training cluster according to claim 3, characterized in that: The function hijacking process of the monitoring hook is as follows: First, save the reference to the original function and define a wrapper function to replace the original implementation; Then, the wrapper function is executed to extract training parameters and status information, calculate performance indicators, and record the data into the performance monitoring system; Finally, call the original function to ensure that the training logic is executed normally.

6. The performance monitoring system for a large model training cluster according to claim 5, characterized in that: The performance metrics include the start time of a single iteration, precise time measurement, and theoretical throughput calculation based on model parameters and batch size.

7. The performance monitoring system for a large model training cluster according to claim 1, characterized in that: The data acquisition module interface access layer is a unified interface for distributed queries, which can realize single-node query and multi-node data merging functions; the multi-node data merging specifically involves: simultaneously acquiring data from the databases of multiple training nodes, and deduplicating, sorting, and merging the query results to ensure the consistency and integrity of the final data.

8. The performance monitoring system for a large model training cluster according to claim 7, characterized in that: When the backend processing module obtains monitoring data from the datasets of multiple training nodes, it adopts an incremental data collection strategy, that is, by maintaining the last updated timestamp of each job, it ensures that only new data since the last collection is obtained.

9. The performance monitoring system for a large model training cluster according to claim 7, characterized in that: When the back-end processing module collects data, it performs coordinated collection of multiple types of data to ensure that different types of data can be updated synchronously, and adopts differentiated collection strategies based on the update frequency characteristics of different data types.

10. The performance monitoring system for a large model training cluster according to claim 1, characterized in that: The composite indicators related to training effectiveness include basic performance indicators and cluster effectiveness indicators; The basic performance indicators include three indicators that reflect the training speed: average iteration time, token processing throughput, and average GPU utilization: The calculation formula of the cluster efficiency index is as follows: MFU = E1 × E2 × E3 E1 = non-fault time / total time E2 = Number of floating-point operations for a complete training session / (GPU nominal power × number of GPUs × total effective training time) E3 = Effective training time / non-failure time Among them, MFU represents the cluster performance indicator, E1 represents the device online availability rate, E2 represents the single-card software stack utilization rate, and E3 represents the model training parallel efficiency.

Images