Method and system for online detection and diagnosis of performance fluctuation of large model hybrid parallel training

Abstract

This invention proposes an online detection and diagnosis method and system for performance fluctuations in large-scale hybrid parallel training, belonging to the field of high-performance computing. The method includes: S1: Deploying probes to monitor iteration steps and asynchronously writing logs; S2: Detecting iteration time using a sliding window, determining performance degradation if conditions are met; S3: Dynamically collecting multi-source heterogeneous performance data from the framework, GPU kernel functions, and hardware layers; S4: Constructing a unified global timeline across nodes and levels based on time anchors and associated identifiers; S5: Semantically calculating key indicators for each dimension according to parallel strategies and identifying inefficient behaviors; S6: Root cause diagnosis to locate abnormal parallel dimensions, nodes, stages, or links; S7: Evaluating the sufficiency of evidence, dynamically adjusting the collection granularity and iterating through S3-S6 if insufficient, until requirements are met or a safety net process is triggered. This invention enables online detection and fine-grained root cause diagnosis of performance fluctuations in large-scale model training with relatively low runtime overhead.

Description

Technical Field

[0001] This invention belongs to the field of high-performance computing, and specifically relates to an online detection and diagnosis method and system for performance fluctuations in large-scale hybrid parallel training. Background Technology

[0002] With the rapid development of large language models, exemplified by the Transformer architecture, generative artificial intelligence technology has made breakthroughs in natural language processing, code generation, and multimodal understanding, demonstrating broad value in academic research and industrial applications. The improvement of large models' capabilities typically relies on the continuous expansion of parameter size, training data scale, and computational load, prompting research institutions and enterprises to build ultra-large-scale distributed training clusters. However, as the training scale continues to expand, the complexity of the training system increases significantly. Modern large model training usually requires the coordinated work of multiple parallel strategies, such as data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism. While these strategies improve computational efficiency and memory utilization, they also introduce complex synchronization boundaries, communication modes, load partitioning methods, and resource contention relationships, making the training process more susceptible to performance fluctuations due to the coupling of multiple hardware and software factors.

[0003] Performance fluctuations refer to the phenomenon of significant performance differences in the same program running under similar load conditions, which is particularly prominent in large-scale GPU clusters and distributed training systems. The root causes can originate from various levels of the software: framework (e.g., uneven pipeline stage partitioning, uneven sequence lengths, Python garbage collection); system software (e.g., operating system interruptions, thread scheduling anomalies); communication system (e.g., NCCL collection communication blocking, RDMA anomalies, network congestion); and hardware (e.g., GPU overheating, HBM memory errors, frequency fluctuations caused by dynamic voltage and frequency regulation, and equipment aging). Because large model training has strong synchronization characteristics, performance degradation in local nodes, local links, or local stages is often amplified through synchronization mechanisms into a global decrease in training throughput, even leading to tailing node problems. Therefore, timely detection and accurate diagnosis of performance fluctuations during large model training are of significant research importance and engineering application value.

[0004] Existing methods for detecting and diagnosing performance fluctuations in large model training generally include the following approaches:

[0005] (1) Offline profiling method based on full performance trajectory acquisition

[0006] These methods typically utilize tools such as Nsight Systems, PyTorch Profiler, or the profiler within the training framework to record operator execution, CUDA kernel functions, communication events, and system resource status with fine granularity during training, and then perform offline analysis after training. These methods provide rich performance details, aiding in the analysis of complex performance issues. However, in large-scale, long-cycle training scenarios, full trajectory acquisition incurs significant computational, storage, and I / O overhead, potentially even disrupting the original training process. Furthermore, offline analysis cannot respond promptly to performance degradation, easily missing the optimal diagnostic window for sudden, random, or short-term performance fluctuations.

[0007] (2) Online monitoring method based on low-overhead coarse-grained index

[0008] These methods typically determine whether training performance has degraded by continuously recording metrics such as training throughput, iteration step time, GPU utilization, memory usage, temperature, and power consumption. They offer advantages such as simple deployment, low operating overhead, and suitability for long-term online operation. However, these methods usually only detect overall training slowdowns and struggle to explain the specific source of the performance degradation. For hybrid parallel training, relying solely on global step time or hardware utilization makes it difficult to determine whether the problem stems from data parallel synchronization waiting, tensor parallel communication overhead, pipeline bubbles, uneven sequence lengths, expert load skew, communication link anomalies, or hardware throttling. Therefore, they are insufficient for fine-grained root cause localization.

[0009] (3) Specialized diagnostic methods based on a single level

[0010] Existing methods address specific levels or problems for optimization and diagnosis. Examples include node health checks for hardware failures, NCCL or RDMA behavior analysis for communication anomalies, uneven pipeline parallelism analysis for uneven stage partitioning, and expert load skew for expert parallelism analysis. While these methods can pinpoint specific types of problems in localized scenarios, performance fluctuations in large-scale hybrid parallel training often exhibit cross-level coupling and cross-parallel dimension propagation characteristics. For instance, local hardware downclocking may increase synchronization wait times within a tensor parallel group, expert load skew may further amplify pipeline bubbles, and communication link congestion may simultaneously affect tensor parallel and expert parallel communication. Therefore, analyzing from only a single level or parallel dimension makes it difficult to establish a global performance view and explain the causal relationships between different inefficiencies.

[0011] In summary, existing technologies either rely on high-overhead full-data collection, making long-term online deployment difficult; or they only collect coarse-grained statistical indicators, which are insufficient to support complex root cause localization; or they lack the ability to perceive multi-dimensional parallel strategies, making it difficult to explain cross-level and cross-dimensional performance fluctuations in hybrid parallel training. Therefore, there is an urgent need for an online performance fluctuation detection and root cause diagnosis method for large-scale model hybrid parallel training. This method should achieve cross-level performance data collection and fusion while ensuring low overhead, combine parallel strategy semantics to model and analyze performance problems, and balance diagnostic timeliness, data collection overhead, and root cause localization accuracy through a dynamic granularity switching mechanism. Summary of the Invention

[0012] To address the aforementioned technical problems, this invention provides an online detection and diagnosis method for performance fluctuations in large-scale hybrid parallel training, comprising the following steps:

[0013] Step S1: Deploy a lightweight online monitoring probe during the large model training process to continuously monitor the training iterations and asynchronously write the monitoring data to the log.

[0014] Step S2: Based on the sliding window, continuously detect the average iteration time of several consecutive steps within the current detection window and the average iteration time of the baseline window. When the performance degradation trigger condition is met, it is determined that training performance has degraded, and then proceed to step S3.

[0015] Step S3: Dynamically trigger fine-grained performance data collection at the training framework layer, GPU kernel function layer, and hardware layer to obtain multi-source heterogeneous performance data within the abnormal time window;

[0016] Step S4: Organize and align the multi-source heterogeneous performance data, and construct a unified global performance timeline across nodes and levels based on the cross-node synchronous communication time anchor point and the association identifier between host-side calls and device-side execution.

[0017] Step S5: Based on the unified global performance timeline, calculate the key performance indicators of each parallel dimension according to the parallel strategy semantics of data parallelism, tensor parallelism, pipeline parallelism and expert parallelism, and identify the corresponding inefficient behaviors.

[0018] Step S6: Perform root cause diagnosis on the key performance indicators and inefficient behaviors to obtain the abnormal parallel dimensions, abnormal nodes, abnormal stages or abnormal links corresponding to the training performance fluctuations, and output the root cause diagnosis results of the performance fluctuations.

[0019] Step S7: Based on the root cause diagnosis results, assess whether the current diagnostic evidence is sufficient. If the evidence is insufficient, dynamically switch the performance data acquisition granularity and execute steps S3 to S6 again until a root cause result that meets the diagnostic requirements is obtained or the backup diagnosis process is triggered.

[0020] Beneficial effects:

[0021] (1) The present invention can build a complete performance analysis data view with low overhead. By combining with the online monitoring triggering mechanism, it can trigger the collection of multi-dimensional performance data of the framework layer, GPU kernel function layer and hardware layer as needed after detecting performance degradation, so that high overhead collection only acts on abnormal time windows, thereby reducing the overall diagnostic overhead in long-term training.

[0022] (2) This invention can perceive the semantics of multi-dimensional parallel strategies for training large models. By combining data parallelism, tensor parallelism, sequence parallelism, pipeline parallelism, expert parallelism and cross-parallel dimension collaboration, it can identify typical inefficient behaviors such as slow nodes, communication blockage, link contention, pipeline bubbles, uneven stage load, expert load skew and hardware jitter.

[0023] (3) The present invention can take into account the timeliness of online diagnosis, diagnostic overhead and root cause localization accuracy. By automatically switching between lightweight monitoring, enhanced diagnosis and in-depth analysis according to the sufficiency of diagnostic evidence, and triggering fixed workload benchmark testing and full analysis guarantee process when necessary, the invention can switch the collection granularity and diagnostic depth as needed, thereby reducing overhead and improving accuracy and timeliness. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the online detection and diagnosis method for performance fluctuations in large-scale hybrid parallel training according to the present invention;

[0025] Figure 2 This is a block diagram of an online detection and diagnosis system for performance fluctuations in large-scale hybrid parallel training of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0027] Example 1

[0028] like Figure 1 As shown in the figure, an online detection and diagnosis method for performance fluctuations in large-scale hybrid parallel training provided by an embodiment of the present invention includes the following steps:

[0029] Step S1: Deploy a lightweight online monitoring probe during the large model training process to continuously monitor the training iterations and asynchronously write the monitoring data to the log.

[0030] Step S2: Based on the sliding window, continuously detect the average iteration time of several consecutive steps within the current detection window and the average iteration time of the baseline window. When the performance degradation trigger condition is met, it is determined that training performance has degraded, and then proceed to step S3.

[0031] Step S3: Dynamically trigger fine-grained performance data collection at the training framework layer, GPU kernel function layer, and hardware layer to obtain multi-source heterogeneous performance data within the abnormal time window;

[0032] Step S4: Organize and align multi-source heterogeneous performance data, and construct a unified global performance timeline across nodes and levels based on cross-node synchronous communication time anchors and the association identifier between host-side calls and device-side executions;

[0033] Step S5: Based on the unified global performance timeline, calculate the key performance indicators of each parallel dimension according to the parallel strategy semantics of data parallelism, tensor parallelism, pipeline parallelism and expert parallelism, and identify the corresponding inefficient behaviors.

[0034] Step S6: Perform root cause diagnosis on key performance indicators and inefficient behaviors to obtain the abnormal parallel dimensions, abnormal nodes, abnormal stages or abnormal links corresponding to training performance fluctuations, and output the root cause diagnosis results of performance fluctuations.

[0035] Step S7: Based on the root cause diagnosis results, assess whether the current diagnostic evidence is sufficient. If the evidence is insufficient, dynamically switch the performance data acquisition granularity and execute steps S3 to S6 again until a root cause result that meets the diagnostic requirements is obtained or the backup diagnosis process is triggered.

[0036] In one embodiment, step S1 above: deploying a lightweight online monitoring probe during the large model training process to continuously monitor the training iterations and asynchronously write the monitoring data to the log, specifically includes:

[0037] This invention adopts a dual-process separation architecture of training process and daemon process: the training process records timestamps at the start and end of each iteration, continuously monitors training iteration steps and hardware performance data, and asynchronously writes the monitoring results to the log; the daemon process continuously reads the monitoring log to provide input for subsequent sliding window detection.

[0038] In one embodiment, step S2 above: Based on a sliding window, continuously detect the average iteration time of several consecutive steps within the current detection window and the average iteration time of the baseline window. When the performance degradation trigger condition is met, it is determined that training performance has degraded, specifically including:

[0039] The daemon process employs a dual-window sliding detection method, comparing the iteration time of several consecutive steps within the current detection window with the average iteration time of the baseline window to determine if there is a persistent decline in training performance.

[0040] Let the training iteration time sequence be... Let n be the total number of iterations. Define a baseline window and a detection window on the training iteration sequence. Let the mean step time within the baseline window be 1. The step time record of the most recent K steps within the detection window is as follows: The performance degradation threshold is The performance degradation trigger condition is:

[0041] , ;

[0042] in, This represents the time taken for the j-th iteration;

[0043] And further satisfy the cooling interval constraint:

[0044] ;

[0045] in, The iteration number corresponding to the last trigger. This represents the number of cooling interval steps.

[0046] In one embodiment, step S3 above: dynamically triggering fine-grained performance data acquisition at the training framework layer, GPU kernel function layer, and hardware layer to obtain multi-source heterogeneous performance data within the abnormal time window, specifically including:

[0047] When a performance degradation is detected, the daemon process writes the acquisition request to the shared signal file. The global master process (rank-0) then reads the request at the start of the next iteration and broadcasts it to other training processes. Upon receiving the broadcast, all training processes synchronously enter fine-grained acquisition mode, continuously acquiring the preset data. ( The system presets the number of iterations (hyperparameters) required for performance data collection, and automatically resumes normal monitoring mode after collection is complete. The collected performance data includes the following three layers of data:

[0048] Step S301: In the performance data acquisition of the training framework layer, hierarchical instrumentation is performed at the training iteration boundaries, forward propagation boundaries, back propagation boundaries, parameter update boundaries, model calculation module boundaries, and communication synchronization primitive boundaries defined by the training framework. Among them, the hierarchical instrumentation adopts the asynchronous time measurement method based on CUDA Events, inserting start events and end events before and after the target operation is executed to obtain the corresponding device execution time. Furthermore, micro-batch numbers, multi-dimensional parallel coordinates, communication data scale, and information on the set of processes participating in the communication are further added to the acquired events. Among them, the multi-dimensional parallel coordinates include at least data parallel coordinates, tensor parallel coordinates, pipeline parallel coordinates, and expert parallel coordinates to establish a correspondence between the semantics of the training phase, the semantics of the parallel topology, and the context information of performance analysis.

[0049] The training side of this invention employs a unified acquisition interface based on CUDA Events to perform layered instrumentation on the framework layer. Framework layer instrumentation is not limited to a single operator or communication operation, but is hierarchically arranged according to training phase semantics, model structure semantics, parallel scheduling semantics, and communication synchronization semantics. This allows the acquisition results to simultaneously support macro-level phase analysis and fine-grained root cause localization. Specifically, instrumentation locations include, but are not limited to, events such as the overall training iteration, forward, backward, optimizer, transformer layer, attention, MLP, moe-router, moe-experts, dp-allreduce, tp-allreduce, ep-alltoall-dispatch, ep-alltoall-combine, p2p-launch, and req.wait(). For clarity, Table 1 provides typical framework layer instrumentation locations and their corresponding semantics.

[0050] Table 1. Typical pile locations in frame structures

[0051]

[0052] The instrumentation method described above corresponds to the unified scope interface in the training code. In actual engineering implementation, the event name and context metadata can be recorded by wrapping the outer layer of the key code section with a scope. The pipeline point-to-point communication initiation section can correspond to the p2p-launch class scope, and the asynchronous communication completion waiting section can correspond to the req.wait() class scope, thereby depicting the communication initiation time and the actual communication completion time, respectively.

[0053] Step S302: Performance data collection at the GPU kernel function layer involves using CUPTI to jointly trace the asynchronous execution behavior of the CUDA Runtime API, CUDA Driver API, and GPU devices. Specifically, the Callback API records the host-side API call timestamps, API names, thread identifiers, and associated identifiers; the Activity API records the device-side kernel function type, device execution start and end times, stream identifier, device identifier, and associated identifiers. Furthermore, by utilizing the shared associated identifiers in the host-side call records and device-side execution records, the host-side call path is matched with the actual execution trajectory on the device side to obtain the corresponding execution relationship, stream-level scheduling relationship, and cross-layer time correspondence between computation kernel functions and communication kernel functions.

[0054] Step S303: At the hardware layer, performance data acquisition involves using NVML to collect GPU utilization, memory utilization, core frequency, base frequency, temperature, power consumption, memory usage, and NVLink throughput, and using PAPI to collect CPU-side performance monitoring unit counters. This invention divides the collected metrics into load inference metrics and performance inference metrics. Load inference metrics are used to characterize changes in program workload, while performance inference metrics are used to characterize the device execution efficiency and its fluctuation state under a given workload, thereby providing a basis for distinguishing between normal load changes and abnormal performance degradation.

[0055] Table 2 Hardware Layer Performance Indicators and Analysis Usage Description

[0056]

[0057] In one embodiment, step S4 above: organizing and aligning multi-source heterogeneous performance data, and constructing a unified global performance timeline across nodes and levels based on cross-node synchronous communication time anchors and the association identifier between host-side calls and device-side executions, specifically includes:

[0058] Step S401: Perform preliminary merging of multi-source heterogeneous performance data in chronological order;

[0059] Step S402: Using AllReduce, AllGather, ReduceScatter, All-to-All and point-to-point communication operations as time anchors for cross-node timing alignment, iteratively calibrate the clock offset between different nodes based on the correspondence between the set of processes participating in the communication, the peer node information, and the communication completion event.

[0060] Step S403: Utilize the same association identifier in the host-side CUDA call record and the device-side kernel function execution record to achieve cross-layer association between framework layer events and kernel function layer events;

[0061] Step S404: Using the time interval of the framework layer event as the time window, filter, aggregate, and mount the kernel function layer record and the hardware layer discrete sampling record to form the kernel function execution sequence and hardware state attributes corresponding to each framework layer event;

[0062] Step S405: Organize the multi-source performance data after cross-node alignment, cross-layer association, and window mounting into a unified global performance timeline.

[0063] In one embodiment, step S5 above involves: based on a unified global performance timeline, calculating key performance indicators for each parallel dimension according to the semantics of parallel strategies such as data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism, and identifying corresponding inefficient behaviors; wherein the key performance indicators include at least any one of the following:

[0064] Step S501: Data parallel step-time balance coefficient, used to characterize the degree of step-time dispersion between different data parallel nodes in the same iteration;

[0065] In a data-parallel scenario, let the step time of the r-th data-parallel node in the i-th iteration be denoted as . Then the step-time balance coefficient across data parallel nodes Defined as:

[0066] ;

[0067] in, This represents the standard deviation of all data nodes in the i-th iteration. This represents the expected value of all data nodes in parallel during the i-th iteration. This represents the step-time balance coefficient across data parallel nodes in the i-th iteration;

[0068] Step S502: Data parallel synchronization point arrival gap, used to characterize the difference in arrival time of each data parallel node on the same synchronization event;

[0069] If the start time of all parallel data nodes on a certain synchronization event is Then the synchronization point reaches the gap Defined as:

[0070] ;

[0071] in, This represents the time when the r-th parallel data node arrives (or begins execution) at a specific synchronization event in the i-th iteration; This represents the time difference between the synchronization points of the data parallel nodes in the i-th iteration;

[0072] Step S503: Data parallel gradient synchronization overhead ratio, used to characterize the proportion of gradient synchronization-related events relative to the total iteration time;

[0073] Let the total duration of gradient synchronization-related events in the i-th iteration be . The i-th iteration step is The proportion of gradient synchronization overhead in the i-th iteration Defined as:

[0074] ;

[0075] Step S504: Data parallel communication overlap ratio, used to characterize the proportion hidden by back-computation in gradient communication time;

[0076] If the time interval between gradient communication and backpropagation overlaps in the i-th iteration is... The total time for gradient communication is Then the communication overlap ratio Defined as:

[0077] ;

[0078] Step S505: Data parallel memory dispersion, used to characterize the difference in peak memory performance between different data parallel nodes;

[0079] If the peak memory usage of each data parallel node is respectively Then the memory dispersion Defined as:

[0080] ;

[0081] in,

[0082] This invention defines step-time equalization coefficient, gradient synchronization overhead ratio, communication overlap ratio, synchronization point arrival gap and memory dispersion to identify slow nodes, gradient synchronization blocking, communication exposure and uneven memory fragmentation problems.

[0083] Step S506: Tensor parallel or sequence parallel communication ratio, used to characterize the time ratio of tensor parallel or sequence parallel related set communication relative to computation;

[0084] In tensor parallelism and sequence parallelism scenarios, let the total communication time of the tensor parallelism or sequence parallelism related sets in the i-th iteration be . The relevant calculation time is Then the proportion of communication Defined as:

[0085] ;

[0086] Step S507: Link utilization, used to characterize the degree of utilization of actual bandwidth relative to theoretical peak bandwidth;

[0087] Let the actual bandwidth measured within a certain communication window be denoted as . The theoretical peak bandwidth is Then the link utilization rate Defined as:

[0088] ;

[0089] Step S508: Kernel function startup overhead ratio, used to characterize the amplification of host-side observation time relative to device-side execution time;

[0090] Let the duration of the wall clock observed on the host side for a certain event be denoted as . The corresponding device-side execution time is Then the kernel function startup overhead is higher than Defined as:

[0091] ;

[0092] Step S509: The arrival gap of synchronization points within the tensor parallel group is used to characterize the difference in arrival time of each member before the start of communication in the same tensor parallel set.

[0093] If the arrival times of each member before the start of parallel set communication for a certain quantity are... The synchronization point within the group reaches the gap. Defined as:

[0094] ;

[0095] in, This represents the arrival time of the r-th process set in the tp-th tensor parallel group of the i-th iteration before the start of communication;

[0096] This invention defines communication ratio, link utilization, kernel function startup overhead ratio, and intra-group synchronization point arrival gap to identify problems such as limited physical bandwidth, excessively fine segmentation granularity, operator fragmentation, and inefficient communication scheduling.

[0097] Step S510: Parallel bubble ratio in the pipeline, used to characterize the idling ratio of the pipeline stage;

[0098] In a pipelined parallel scenario, let the active time of a certain PP stage in the i-th iteration be . The total iteration time is Then the bubble ratio Defined as:

[0099] ;

[0100] Step S511: Pipeline stage load unevenness coefficient, used to characterize the difference in average computation time between different pipeline stages;

[0101] Let the average computation time of the s-th PP stage be... Then the stage load unevenness coefficient Defined as:

[0102] ;

[0103] Step S512: Pipeline parallel forward computation time model for variable-length sequence training, used to characterize the correspondence between the computation time of the forward stage and the total number of lexical units and the sum of squares of sequence length in the variable-length sequence training scenario; construct a sample set based on the forward computation time model, and calculate the Pearson correlation coefficient and Spearman correlation coefficient to represent the linear correlation and monotonic correlation between the computation time of the forward stage and the total number of lexical units and the sum of squares of sequence length, respectively;

[0104] For training scenarios involving variable-length sequences, let the computation time T of the forward phase satisfy the following approximate model:

[0105] ;

[0106] in, Indicates the total number of lexical elements. This represents the sequence length of the i-th micro-batch. Represents the sum of squares of the sequence length. This represents the weight of linear computation time, which is proportional to the total number of lexical units. The weight represents the computation time required for quadratic operations, which is proportional to the square of the sequence length. This represents system and hardware noise.

[0107] For the steady-state training phase of the same rank, extract all forward phase sample points, and construct a sample set after removing the warm-up phase iterations:

[0108] ;

[0109] Where M represents the total number of iterations required for analysis. This represents the data samples from iteration number i=1 to M. This represents the total number of lexical units in the i-th iteration. This represents the sum of squares of the sequence lengths in the i-th iteration. This represents the forward computation time of the i-th iteration;

[0110] Then, the Pearson correlation coefficients were calculated:

[0111] ;

[0112] And the Spearman correlation coefficient:

[0113] ;

[0114] Where x represents the input feature variable, specifically the total number of tokens N or the sum of squares of the sequence length. , This represents the arithmetic mean of the input feature variables. This indicates the time taken for the forward phase. represents the arithmetic mean of the time spent in the forward phase, and rank represents the rank of the feature variables;

[0115] Further results were obtained:

[0116] ;

[0117] ;

[0118] ;

[0119] ;

[0120] in, and Let T represent the Pearson correlation coefficient and Spearman correlation coefficient, respectively, between the forward phase time T and the total number of tokens N. and These represent the forward phase time T and the sum of squares of the sequence length, respectively. The Pearson correlation coefficient and the Spearman correlation coefficient between them;

[0121] Step S513: Coefficient of variation of forward phase time , used to characterize the degree of dispersion in the forward computation time of the same rank in the steady state stage;

[0122] ;

[0123] in, This represents the standard deviation of the forward computation time during the steady-state phase in the i-th iteration. This represents the mathematical expectation of the forward computation time during the steady-state phase in the i-th iteration.

[0124] Step S514: Point-to-point effective bandwidth and point-to-point communication blocking time are used to characterize the actual point-to-point transmission efficiency and dependency waiting overhead.

[0125] For a single point-to-point transmission, if the data size is B, the upper bound of the actual transmission time is... The total waiting time at the receiving end is Then the effective bandwidth of point-to-point and point-to-point communication blocking time Defined as:

[0126] ;

[0127] ;

[0128] Step S515: Hardware jitter amplification ratio, used to identify hardware jitter or abnormal frequency reduction after controlling for load factors;

[0129] For events with the same load characteristics, if the observation duration is Historical benchmark Then the hardware jitter amplification ratio Defined as:

[0130] ;

[0131] This invention defines the above-mentioned bubble ratio, stage load unevenness coefficient, variable length sequence forward computation time model, Pearson correlation coefficient, Spearman correlation coefficient, forward stage time variation coefficient, point-to-point effective bandwidth, point-to-point communication blocking time and hardware jitter amplification ratio, which are used to identify pipeline stage division unevenness, input load driven fluctuations, abnormal load unevenness, stage dependency waiting and hardware jitter problems.

[0132] Step S516: Expert load variation coefficient, used to characterize the degree of dispersion of word load distribution among experts in the expert parallel scenario;

[0133] In a parallel expert scenario, let the number of lexical units received by each expert in a certain layer during a certain iteration be . Then the expert load variation coefficient Defined as:

[0134] ;

[0135] Step S517: Top Expert Percentage, used to represent the proportion of the most popular experts in the total word units;

[0136] If the expert who receives the most word elements has the following number of word elements... The total number of lexical elements is The proportion of top experts Defined as:

[0137] ;

[0138] Step S518: Expert parallel rank-level load dispersion, used to characterize the difference in total word load undertaken by different expert parallel ranks;

[0139] Let the total number of word units handled by the r-th expert in parallel ranking be . Then the rank-level load dispersion is defined as:

[0140] ;

[0141] in, Indicates standard deviation, Represents the mathematical expectation;

[0142] Step S519: The proportion of expert parallel communication and the proportion of communication overlap are used to characterize the overhead of expert parallel communication and the degree to which it is hidden by computation;

[0143] Let the pure communication times of dispatch and combine be respectively and The MoE window is The proportion of expert parallel communication Defined as:

[0144] ;

[0145] in, This indicates the total time from when the framework initiates the MoE dispatch operation to when the dispatch communication is completed. This represents the total time calculated by experts. This indicates the total time from when the framework initiates the MoE combine operation to when the combine communication is completed;

[0146] If the time of overlap between expert parallel communication and expert computation is The expert parallel communication time is The overlap ratio of expert parallel communication Defined as:

[0147] ;

[0148] Step S520: Term drop rate and route entropy, used to characterize term drop phenomenon and the degree of concentration of route distribution under capacity-constrained conditions;

[0149] If the number of discarded lexics is The total number of lexical units routed to the expert is Then the word discard rate Defined as:

[0150] ;

[0151] Furthermore, set Then the routing entropy Defined as:

[0152] ;

[0153] in, To prevent the occurrence of extremely small constants with logarithmic zero values ​​during numerical calculations, E represents the total number of routable experts in the current routing layer, and e represents the index of the e-th expert. .

[0154] This invention defines the expert load variation coefficient, the proportion of top experts, the rank-level load dispersion, the proportion of expert parallel communication, the communication overlap ratio, the term discard rate, and the routing entropy to identify expert load skew, hot expert overload, all-to-all communication bottleneck, capacity limitation, and routing collapse or routing anomaly issues.

[0155] Step S521: Cross-parallel dimension collaborative performance metrics, including EP-PP bubble coupling coefficient, EP-TP communication contention ratio, and PP bubble utilization rate, which are used to characterize the coupling relationship between expert load skew and pipeline bubbles, the degree of shared link contention between expert parallel communication and tensor parallel communication, and the degree to which pipeline bubble intervals are utilized by other communications.

[0156] In cross-parallel dimension collaboration scenarios, let the expert load skew sequence be... The bubble ratio sequence of the production line is as follows: The EP-PP bubble coupling coefficient is defined as follows:

[0157] ;

[0158] Suppose that the overlap length between the expert parallel communication interval and the tensor parallel communication interval in a certain iteration is . The total duration of the two types of communication are respectively and The EP-TP communication contention ratio is defined as follows:

[0159] ;

[0160] Let the time used by other communications within the bubble section of the pipeline be . The total bubble time is The PP bubble utilization rate is defined as:

[0161] ;

[0162] This invention defines a cross-parallel dimension collaboration metric to identify inefficient propagation paths between different parallel dimensions.

[0163] This invention, through the aforementioned key performance indicators, can identify inefficient behaviors such as slow nodes, gradient synchronization blocking, low link utilization, amplified kernel function startup overhead, operator fragmentation, pipeline bubble anomaly, uneven stage load, point-to-point communication blocking, normal load fluctuations and abnormal hardware jitter under variable-length sequence training, expert load skew, hotspot expert overload, all-to-all communication bottleneck, term discarding, and cross-parallel dimension resource contention.

[0164] In one embodiment, step S6 above: performing root cause diagnosis on key performance indicators and inefficient behaviors to obtain the abnormal parallel dimensions, abnormal nodes, abnormal stages, or abnormal links corresponding to training performance fluctuations, and outputting the root cause diagnosis results of performance fluctuations, specifically includes:

[0165] Step S601: In the data parallel domain, when the synchronization point arrival gap increases and the step balance coefficient increases, the node with the latest start time is identified as the slow node, and the GPU temperature, core frequency, and power consumption are queried within the dynamic time window corresponding to that node; when the temperature is consistently high and the average frequency is lower than the base frequency, it is determined that the performance fluctuation is caused by thermal throttling or frequency reduction; otherwise, combined with the memory dispersion and communication overlap ratio, it is determined that the performance fluctuation is caused by uneven load, data fragmentation difference, uneven memory fragmentation, or difference in data preparation efficiency.

[0166] In data parallel scenarios, the system first aggregates the synchronization events of all nodes within the same data parallel group according to the iteration number and synchronization event name, and calculates the arrival gap of the synchronization point. The node with the latest start time is identified as the slow node in this synchronization event. Subsequently, a dynamic diagnostic window is constructed based on the slow node's start time in the current iteration to the synchronization start time, and GPU temperature, core frequency, power consumption, GPU utilization, and memory status are queried within the window. When the temperature consistently approaches the temperature control threshold and the average frequency is lower than the base frequency, it is determined to be a performance fluctuation caused by thermal throttling or frequency reduction; when… Continue to rise and When the value remains consistently low, it is determined to be a performance fluctuation caused by insufficient gradient synchronization exposure or overlap; when When the increase is continuous, it is determined to be a performance fluctuation caused by uneven memory partitioning, memory fragmentation, or differences in data batch distribution; when the above hardware indicators do not show significant abnormalities, and When the load continues to rise, it is determined to be a performance fluctuation caused by uneven load, differences in data preparation efficiency, or abnormal preceding computation paths of slow nodes.

[0167] Step S602: In the tensor parallel or sequence parallel domain, when the proportion of tensor parallel communication is high and the link utilization is close to the peak, it is determined to be a performance fluctuation caused by physical bandwidth limitation; when the proportion of tensor parallel communication is high and the link utilization is lower than the preset threshold, combined with the kernel function startup overhead ratio and the arrival difference of synchronization points within the group, it is determined to be a performance fluctuation caused by communication block granularity being too small, segmentation being too fine, operator fragmentation, communication path being too long, or communication scheduling being inefficient.

[0168] In tensor parallelism and sequence parallelism scenarios, the first step is to aggregate ensemble communication events and related computation events within the same layer around the same tensor parallelism group, and then perform computation. , , and .when Continue to rise and When the performance approaches the theoretical peak, it is determined to be a performance fluctuation caused by physical bandwidth limitation or link saturation; when higher and When it is too low, combine and Further analysis, if Obviously too large and If the value continues to rise, it is determined that the performance fluctuation is caused by excessively fine segmentation granularity, too many short kernel functions, and CPU submission overhead. If the value continues to rise, it is determined that the performance fluctuation is caused by uneven arrival of members within the group, uneven execution of preceding operators, or uneven aggregate communication scheduling. If the link utilization and the gap in arrival within the group are both abnormal, it is determined that the performance fluctuation is caused by excessively long communication paths, improper coupling between communication and computing scheduling, or cross-device resource contention.

[0169] Step S603: In the pipeline parallel domain, when the bubble ratio is higher than the preset threshold and the stage load unevenness coefficient continues to increase, it is determined that the pipeline stage division is uneven; for variable length sequence training scenarios, if the forward stage time is highly correlated with the total number of tokens or the sum of squares of sequence length, it is determined that the fluctuation of the forward stage time is mainly caused by the change of input load; if the correlation is significantly low and the coefficient of variation of the forward stage time is consistently high, it is determined that the performance fluctuation is caused by abnormal load unevenness, hardware jitter, abnormal execution path, or local resource contention, combined with the hardware jitter amplification ratio; when the point-to-point communication blocking time is higher than the preset threshold, it is determined that the upstream and downstream stages depend on waiting or the point-to-point communication is blocked.

[0170] In a pipelined parallel scenario, the system first performs calculations on each stage. and If the bubble ratio increases and the stage load unevenness coefficient continues to increase, it is determined that the performance fluctuation is caused by uneven static layer partitioning or excessive computational load in some stages. For variable-length sequence training scenarios, the system further optimizes based on the sample set. calculate as well as When at least one set of correlation coefficients remains at a high level, it is determined that fluctuations in the forward propagation phase time are mainly caused by changes in input load; when the correlation is significantly low... When it remains persistently high, combined with Further analysis reveals that if the slowdown ratio is significantly high, it is determined to be performance fluctuation caused by hardware jitter, abnormal frequency reduction, or local resource contention; otherwise, it is determined to be an abnormal execution path or abnormal load unevenness that is difficult to explain by normal load. For point-to-point communication, if Continuously above the threshold and If no significant decrease is observed, it is determined to be a dependency waiting between upstream and downstream stages; if At the same time as rising If the value decreases, it is determined to be point-to-point link congestion or point-to-point communication blockage;

[0171] Step S604: In the expert parallel domain, when any of the following is abnormal: expert load variation coefficient, top expert ratio, rank-level load dispersion, term discard rate, or routing entropy, the performance fluctuation is determined by combining the expert parallel communication ratio and communication overlap ratio, which is caused by expert load skew, hotspot expert overload, router collapse, all-to-all communication bottleneck, capacity limitation, or routing anomaly.

[0172] In expert parallel scenarios, the system first collects statistics on the word reception status of each expert, dispatch and combine communication events, and word dropping events around the same MoE layer, and then calculates... , , , , , and .when and When the load continues to rise, it is determined to be either an expert load imbalance or an overload of hotspot experts; when When the load continues to rise, it is determined that the load distribution among the parallel ranks of experts is uneven; when Continue to rise and When the value remains consistently low, it is determined to be a performance fluctuation caused by exposure to All-to-All communication, insufficient overlap, or contention for shared links; when When the level rises, it is determined that the expert capacity is limited or the routing distribution is abnormal; when When the performance continuously decreases and the routing is concentrated among a few experts, it is determined to be a performance fluctuation caused by route collapse, router collapse, or abnormal router training state.

[0173] Step 605: In the scenario of cross-parallel dimension collaborative analysis, when the EP-PP bubble coupling coefficient continues to increase, it is determined that the expert load skew has an amplifying effect on the pipeline bubble; when the EP-TP communication competition ratio increases and the link utilization is close to the peak, it is determined that there is significant competition between expert parallel communication and tensor parallel communication on the shared link; when the PP bubble utilization is consistently low, it is determined that the pipeline idle time is not effectively utilized by communication in other parallel dimensions, resulting in communication exposure and resource idleness.

[0174] In cross-parallel dimension collaboration scenarios, the system further analyzes the cascading relationships between various parallel dimensions. When When the load continues to rise, it is determined that the expert load skew has an amplifying effect on the bubbles in the production line; when When the link utilization rate continues to rise and approaches its peak, it is determined that there is significant competition between expert parallel communication and tensor parallel communication on the shared link; when When the value remains consistently low, it indicates that the pipeline idle time is not being effectively utilized by communication in other parallel dimensions, resulting in cross-dimensional serialization, resource idleness, or scheduling mismatch issues.

[0175] Through the above-described hierarchical diagnostic process, the following can be output: abnormal parallel dimension, abnormal node, abnormal stage, abnormal expert, abnormal link and its corresponding root cause label.

[0176] In one embodiment, step S7 above: Based on the root cause diagnosis results, assess whether the current diagnostic evidence is sufficient. If the evidence is insufficient, dynamically switch the performance data acquisition granularity and execute steps S3 to S6 again until a root cause result that meets the diagnostic requirements is obtained or the backup diagnostic process is triggered, including the following steps:

[0177] Step 701: Divide the performance data acquisition granularity into three levels. The first level is fine-grained acquisition at the framework layer and sampling at the hardware layer. The second level adds some communication kernel function tracing on the basis of the first level. The third level adds full kernel function tracing and CPU-side performance counter acquisition on the basis of the second level.

[0178] Step 702: Based on the current diagnostic results, assess the adequacy of the diagnosis from four dimensions: root cause localization accuracy, chain of evidence completeness, cross-dimensional consistency, and data coverage.

[0179] Step 703: When the evaluation results indicate that the current diagnosis is insufficient, automatically upgrade to a higher level of data collection granularity and re-execute steps S3 to S6;

[0180] Step 704: If the root cause cannot be fully located even at the highest level of performance data collection granularity, trigger the fixed workload benchmark test and full analysis as a safety net process;

[0181] In the fixed workload benchmark test, let the average execution time of node r on the b-th benchmark test be... The minimum execution time among all nodes is Then the performance degradation coefficient of node r on the b-th benchmark is:

[0182] ;

[0183] when When the degradation threshold is exceeded, the node is marked as abnormal on the benchmark test. Furthermore, a comprehensive abnormal score is calculated based on the results of each benchmark test to identify abnormal nodes with hardware performance degradation. The benchmark tests include at least one of the following: computational benchmark tests, aggregated communication benchmark tests, and point-to-point communication benchmark tests. These tests are used to further confirm the computational capability degradation, communication capability degradation, or link anomalies of suspected nodes under the condition that the control workload is basically consistent.

[0184] Step 705: Based on the operational overhead, evidence completeness, and root cause localization accuracy corresponding to different performance data acquisition granularities, switch between continuous online monitoring, enhanced diagnosis, and in-depth analysis to achieve a balance between acquisition overhead and diagnostic accuracy.

[0185] Example 2

[0186] like Figure 2 As shown, this embodiment of the invention provides an online detection and diagnosis system for performance fluctuations in large-scale hybrid parallel training, comprising the following modules:

[0187] The data acquisition module 81 is used to deploy a lightweight online monitoring probe during the large model training process, continuously monitor the training iteration steps, and asynchronously write the monitoring data to the log.

[0188] The monitoring module 82 is used to continuously detect the average iteration time of several consecutive steps within the current detection window and the baseline window based on a sliding window. When the performance degradation trigger condition is met, it is determined that training performance has degraded, and then the process is switched to the fine-grained performance data acquisition module.

[0189] The fine-grained performance data acquisition module 83 is used to dynamically trigger fine-grained performance data acquisition at the training framework layer, GPU kernel function layer and hardware layer to obtain multi-source heterogeneous performance data within the abnormal time window.

[0190] The data aggregation and alignment module 84 is used to organize and align multi-source heterogeneous performance data. Based on the cross-node synchronous communication time anchor point and the association identifier between host-side calls and device-side execution, it constructs a unified global performance timeline across nodes and levels.

[0191] The parallel strategy semantic analysis module 85 is used to calculate the key performance indicators of each parallel dimension based on the unified global performance timeline and according to the parallel strategy semantics of data parallelism, tensor parallelism, pipeline parallelism and expert parallelism, and to identify the corresponding inefficient behaviors.

[0192] The root cause diagnosis module 86 is used to perform root cause diagnosis on key performance indicators and inefficient behaviors to obtain the abnormal parallel dimensions, abnormal nodes, abnormal stages or abnormal links corresponding to training performance fluctuations, and output the root cause diagnosis results of performance fluctuations.

[0193] Result Output Module 87: Used to assess whether the current diagnostic evidence is sufficient based on the root cause diagnosis results. When the evidence is insufficient, it dynamically switches the performance data acquisition granularity and executes the fine-grained performance data acquisition module, data aggregation and alignment module, parallel strategy semantic analysis module and root cause diagnosis module again until a root cause result that meets the diagnostic requirements is obtained or the backup diagnosis process is triggered.

[0194] An online detection and diagnosis device for performance fluctuations in large-scale hybrid parallel training includes one or more electronic devices, wherein the one or more electronic devices are used to implement the online detection and diagnosis method for performance fluctuations in large-scale hybrid parallel training.

[0195] An electronic device includes: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors enable the one or more processors to implement an online detection and diagnosis method for performance fluctuations in large-scale hybrid parallel training.

[0196] A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement an online detection and diagnosis method for performance fluctuations in large-scale hybrid parallel training.

[0197] A non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements an online detection and diagnosis method for performance fluctuations in large-scale hybrid parallel training.

[0198] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for online detection and diagnosis of performance fluctuations in large-scale model hybrid parallel training, characterized in that, include: Step S1: Deploy a lightweight online monitoring probe during the large model training process to continuously monitor the training iterations and asynchronously write the monitoring data to the log. Step S2: Based on the sliding window, continuously detect the average iteration time of several consecutive steps within the current detection window and the average iteration time of the baseline window. When the performance degradation trigger condition is met, it is determined that training performance has degraded, and then proceed to step S3. Step S3: Dynamically trigger fine-grained performance data collection at the training framework layer, GPU kernel function layer, and hardware layer to obtain multi-source heterogeneous performance data within the abnormal time window; Step S4: Organize and align the multi-source heterogeneous performance data, and construct a unified global performance timeline across nodes and levels based on the cross-node synchronous communication time anchor point and the association identifier between host-side calls and device-side execution. Step S5: Based on the unified global performance timeline, calculate the key performance indicators of each parallel dimension according to the parallel strategy semantics of data parallelism, tensor parallelism, pipeline parallelism and expert parallelism, and identify the corresponding inefficient behaviors. Step S6: Perform root cause diagnosis on the key performance indicators and inefficient behaviors to obtain the abnormal parallel dimensions, abnormal nodes, abnormal stages or abnormal links corresponding to the training performance fluctuations, and output the root cause diagnosis results of the performance fluctuations. Step S7: Based on the root cause diagnosis results, assess whether the current diagnostic evidence is sufficient. If the evidence is insufficient, dynamically switch the performance data acquisition granularity and execute steps S3 to S6 again until a root cause result that meets the diagnostic requirements is obtained or the backup diagnosis process is triggered.

2. The online detection and diagnosis method for performance fluctuations in large-scale model hybrid parallel training according to claim 1, characterized in that, Step S2: Based on a sliding window, continuously detect the average iteration time of several consecutive steps within the current detection window and the average iteration time of the baseline window. When the performance degradation trigger condition is met, it is determined that training performance has degraded. Specifically, this includes: Using a dual-window sliding detection method, let the training iteration time sequence be... Let n be the total number of iterations. A baseline window and a detection window are defined on the training iteration time sequence. Let the average step time within the baseline window be... The step time record of the most recent K steps within the detection window is as follows: The performance degradation threshold is The performance degradation trigger condition is: ， ； in, This represents the time taken for the j-th iteration; And further satisfy the cooling interval constraint: ； in, The iteration number corresponding to the last trigger. This represents the number of cooling interval steps.

3. The online detection and diagnosis method for performance fluctuations in large-scale model hybrid parallel training according to claim 2, characterized in that, Step S3: Dynamically trigger fine-grained performance data acquisition at the training framework layer, GPU kernel function layer, and hardware layer to obtain multi-source heterogeneous performance data within the abnormal time window, specifically including: Step S301: The performance data acquisition at the training framework layer involves performing layered instrumentation at the training iteration boundaries, forward propagation boundaries, back propagation boundaries, parameter update boundaries, model computation module boundaries, and communication synchronization primitive boundaries defined by the training framework. Step S302: Performance data collection at the GPU kernel function layer involves using CUPTI to jointly trace the CUDA Runtime API, CUDA Driver API, and asynchronous execution behavior of the GPU device; Step S303: At the hardware level, performance data acquisition involves using NVML to collect GPU utilization, memory utilization, core frequency, base frequency, temperature, power consumption, memory usage, and NVLink throughput, and using PAPI to collect CPU-side performance monitoring unit counters.

4. The online detection and diagnosis method for performance fluctuations in large-scale model hybrid parallel training according to claim 3, characterized in that, Step S4: Organize and align the multi-source heterogeneous performance data, and construct a unified global performance timeline across nodes and levels based on cross-node synchronous communication time anchors and the association identifiers between host-side calls and device-side executions. Specifically, this includes: Step S401: Initially merge the multi-source heterogeneous performance data in chronological order; Step S402: Using AllReduce, AllGather, ReduceScatter, All-to-All and point-to-point communication operations as time anchors for cross-node timing alignment, iteratively calibrate the clock offset between different nodes based on the correspondence between the set of processes participating in the communication, the peer node information, and the communication completion event. Step S403: Utilize the same association identifier in the host-side CUDA call record and the device-side kernel function execution record to achieve cross-layer association between framework layer events and kernel function layer events; Step S404: Using the time interval of the framework layer event as the time window, filter, aggregate, and mount the kernel function layer record and the hardware layer discrete sampling record to form the kernel function execution sequence and hardware state attributes corresponding to each framework layer event; Step S405: Organize the multi-source performance data after cross-node alignment, cross-layer association, and window mounting into a unified global performance timeline.

5. The online detection and diagnosis method for performance fluctuations in large-scale hybrid parallel training according to claim 4, characterized in that, The key performance indicators of step S5 specifically include: Step S501: Data parallel step-time balance coefficient, used to characterize the degree of step-time dispersion between different data parallel nodes in the same iteration; Step S502: Data parallel synchronization point arrival gap, used to characterize the difference in arrival time of each data parallel node on the same synchronization event; Step S503: Data parallel gradient synchronization overhead ratio, used to characterize the proportion of gradient synchronization-related events relative to the total iteration time; Step S504: Data parallel communication overlap ratio, used to characterize the proportion hidden by back-computation in gradient communication time; Step S505: Data parallel memory dispersion, used to characterize the difference in peak memory performance between different data parallel nodes; Step S506: Tensor parallel or sequence parallel communication ratio, used to characterize the time ratio of tensor parallel or sequence parallel related set communication relative to computation; Step S507: Link utilization, used to characterize the degree of utilization of actual bandwidth relative to theoretical peak bandwidth; Step S508: Kernel function startup overhead ratio, used to characterize the amplification of host-side observation time relative to device-side execution time; Step S509: The arrival gap of synchronization points within the tensor parallel group is used to characterize the difference in arrival time of each member before the start of communication in the same tensor parallel set. Step S510: Parallel bubble ratio in the pipeline, used to characterize the idling ratio of the pipeline stage; Step S511: Pipeline stage load unevenness coefficient, used to characterize the difference in average computation time between different pipeline stages; Step S512: A pipelined parallel forward computation time model for variable-length sequence training is used to characterize the relationship between the computation time of the forward stage and the total number of lexical units and the sum of squares of the sequence length in the variable-length sequence training scenario; a sample set is constructed based on the forward computation time model, and the Pearson correlation coefficient and Spearman correlation coefficient are calculated to represent the linear correlation and monotonic correlation between the computation time of the forward stage and the total number of lexical units and the sum of squares of the sequence length, respectively; Step S513: Forward stage time variation coefficient, used to characterize the dispersion of the forward computation time of the same rank in the steady state stage; Step S514: Point-to-point effective bandwidth and point-to-point communication blocking time are used to characterize the actual point-to-point transmission efficiency and dependency waiting overhead. Step S515: Hardware jitter amplification ratio, used to identify hardware jitter or abnormal frequency reduction after controlling for load factors; Step S516: Expert load variation coefficient, used to characterize the degree of dispersion of word load distribution among experts in the expert parallel scenario; Step S517: Top Expert Percentage, used to represent the proportion of the most popular experts in the total word units; Step S518: Expert parallel rank-level load dispersion, used to characterize the difference in total word load undertaken by different expert parallel ranks; Step S519: The proportion of expert parallel communication and the proportion of communication overlap are used to characterize the overhead of expert parallel communication and the degree to which it is hidden by computation; Step S520: Term drop rate and route entropy, used to characterize term drop phenomenon and the degree of concentration of route distribution under capacity-constrained conditions; Step S521: Cross-parallel dimension collaborative performance metrics, including EP-PP bubble coupling coefficient, EP-TP communication contention ratio, and PP bubble utilization rate, which are used to characterize the coupling relationship between expert load skew and pipeline bubbles, the degree of shared link contention between expert parallel communication and tensor parallel communication, and the degree to which pipeline bubble intervals are utilized by other communications.

6. The online detection and diagnosis method for performance fluctuations in large-scale model hybrid parallel training according to claim 5, characterized in that, Step S6: Perform root cause diagnosis on the key performance indicators and inefficient behaviors to obtain the abnormal parallel dimensions, abnormal nodes, abnormal stages, or abnormal links corresponding to the training performance fluctuations, and output the root cause diagnosis results of the performance fluctuations, specifically including: Step S601: In the data parallel domain, when the synchronization point arrival gap increases and the step balance coefficient increases, the node with the latest start time is identified as the slow node, and the GPU temperature, core frequency, and power consumption are queried within the dynamic time window corresponding to that node; when the temperature is consistently high and the average frequency is lower than the base frequency, it is determined that the performance fluctuation is caused by thermal throttling or frequency reduction; otherwise, combined with the memory dispersion and communication overlap ratio, it is determined that the performance fluctuation is caused by uneven load, data fragmentation difference, uneven memory fragmentation, or difference in data preparation efficiency. Step S602: In the tensor parallel or sequence parallel domain, when the proportion of tensor parallel communication is high and the link utilization is close to the peak, it is determined to be a performance fluctuation caused by physical bandwidth limitation; when the proportion of tensor parallel communication is high and the link utilization is lower than the preset threshold, combined with the kernel function startup overhead ratio and the arrival difference of synchronization points within the group, it is determined to be a performance fluctuation caused by communication block granularity being too small, segmentation being too fine, operator fragmentation, communication path being too long, or communication scheduling being inefficient. Step S603: In the pipeline parallel domain, when the bubble ratio is higher than the preset threshold and the stage load unevenness coefficient continues to increase, it is determined that the pipeline stage division is uneven; for variable length sequence training scenarios, if the forward stage time is highly correlated with the total number of tokens or the sum of squares of sequence length, it is determined that the fluctuation of the forward stage time is mainly caused by the change of input load; if the correlation is significantly low and the coefficient of variation of the forward stage time is consistently high, it is determined that the performance fluctuation is caused by abnormal load unevenness, hardware jitter, abnormal execution path, or local resource contention, combined with the hardware jitter amplification ratio; when the point-to-point communication blocking time is higher than the preset threshold, it is determined that the upstream and downstream stages depend on waiting or the point-to-point communication is blocked. Step S604: In the expert parallel domain, when any of the following is abnormal: expert load variation coefficient, top expert ratio, rank-level load dispersion, term discard rate, or routing entropy, the performance fluctuation is determined by combining the expert parallel communication ratio and communication overlap ratio, which is caused by expert load skew, hotspot expert overload, router collapse, all-to-all communication bottleneck, capacity limitation, or routing anomaly. Step 605: In the scenario of cross-parallel dimension collaborative analysis, when the EP-PP bubble coupling coefficient continues to increase, it is determined that the expert load skew has an amplifying effect on the pipeline bubble; when the EP-TP communication competition ratio increases and the link utilization is close to the peak, it is determined that there is significant competition between expert parallel communication and tensor parallel communication on the shared link; when the PP bubble utilization is consistently low, it is determined that the pipeline idle time is not effectively utilized by communication in other parallel dimensions, resulting in communication exposure and resource idleness.

7. The online detection and diagnosis method for performance fluctuations in large-scale model hybrid parallel training according to claim 6, characterized in that, Step S7: Based on the root cause diagnosis results, assess whether the current diagnostic evidence is sufficient. If the evidence is insufficient, dynamically switch the performance data acquisition granularity and execute steps S3 to S6 again until a root cause result that meets the diagnostic requirements is obtained or the backup diagnostic process is triggered, including the following steps: Step 701: Divide the performance data acquisition granularity into three levels. The first level is fine-grained acquisition at the framework layer and sampling at the hardware layer. The second level adds some communication kernel function tracing on the basis of the first level. The third level adds full kernel function tracing and CPU-side performance counter acquisition on the basis of the second level. Step 702: Based on the current diagnostic results, assess the adequacy of the diagnosis from four dimensions: root cause localization accuracy, chain of evidence completeness, cross-dimensional consistency, and data coverage. Step 703: When the evaluation results indicate that the current diagnosis is insufficient, automatically upgrade to a higher level of data collection granularity and re-execute steps S3 to S6; Step 704: If the root cause cannot be fully located even at the highest level of performance data collection granularity, trigger the fixed workload benchmark test and full analysis as a safety net process; Step 705: Based on the operational overhead, evidence completeness, and root cause localization accuracy corresponding to different performance data acquisition granularities, switch between continuous online monitoring, enhanced diagnosis, and in-depth analysis to achieve a balance between acquisition overhead and diagnostic accuracy.

8. An online detection and diagnosis system for performance fluctuations in large-scale hybrid parallel training, characterized in that, Includes the following modules: The data acquisition module is used to deploy lightweight online monitoring probes during the training process of large models, continuously monitor the training iterations, and asynchronously write the monitoring data to the log. The monitoring module is used to continuously detect the average iteration time of several consecutive steps within the current detection window and the baseline window based on a sliding window. When the performance degradation trigger condition is met, it is determined that training performance has degraded, and then the process is switched to the fine-grained performance data acquisition module. The fine-grained performance data acquisition module is used to dynamically trigger fine-grained performance data acquisition at the training framework layer, GPU kernel function layer, and hardware layer to obtain multi-source heterogeneous performance data within abnormal time windows. The data aggregation and alignment module is used to organize and align the multi-source heterogeneous performance data. Based on the cross-node synchronous communication time anchor point and the association identifier between host-side calls and device-side execution, a unified global performance timeline is constructed across nodes and levels. The parallel strategy semantic analysis module is used to calculate the key performance indicators of each parallel dimension based on the unified global performance timeline and according to the parallel strategy semantics of data parallelism, tensor parallelism, pipeline parallelism and expert parallelism, and to identify the corresponding inefficient behaviors. The root cause diagnosis module is used to perform root cause diagnosis on the key performance indicators and inefficient behaviors to obtain the abnormal parallel dimensions, abnormal nodes, abnormal stages or abnormal links corresponding to the training performance fluctuations, and output the root cause diagnosis results of the performance fluctuations. The results output module is used to assess whether the current diagnostic evidence is sufficient based on the root cause diagnosis results. When the evidence is insufficient, it dynamically switches the performance data acquisition granularity and executes the fine-grained performance data acquisition module, data aggregation and alignment module, parallel strategy semantic analysis module and root cause diagnosis module again until a root cause result that meets the diagnostic requirements is obtained or the backup diagnosis process is triggered.

9. An online detection and diagnosis device for performance fluctuations in large-scale hybrid parallel training, characterized in that, It includes one or more electronic devices, wherein the one or more electronic devices are used to implement the method of any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 7.

Images