Model training hotspot identification method and system combining cross-thread dependence and multi-critical path analysis, and application

Abstract

The invention discloses a model training hotspot identification method combining cross-thread dependence and multi-critical path analysis, and the method comprises the steps: 1, collecting an event execution track in a model training process, and carrying out the analysis and standardization of collected original performance data into a unified format; 2, aggregating the original events through hierarchical modeling, and constructing edge connection based on a dependency relationship to form a calculation dependency graph; step 3, identifying a decisive event sequence and a critical path in a model training process based on the constructed calculation dependency graph; and 4, through overlap analysis and statistical processing, extracting hot events with actual optimization values from the key paths, and generating a sorting list. The invention further discloses a system for implementing the method, and the system has a wide application prospect.

Description

Technical Field

[0001] This invention relates to computer technology, and more particularly to the fields of artificial intelligence and performance analysis and optimization. Specifically, it relates to a method, system and application for identifying hotspots in model training that combines cross-thread dependency modeling and multi-critical-path analysis. Background Technology

[0002] With the continuous growth in the size of large models (such as large language models, LLMs), performance optimization of their training process has become a key issue in deep learning system research. A typical training process involves close collaboration between the central processing unit (CPU) and the graphics processing unit (GPU), covering multiple computationally intensive stages such as forward propagation, back propagation, and parameter updates.

[0003] To maximize resource utilization, mainstream deep learning frameworks commonly employ multi-threaded scheduling and multi-CUDA stream mechanisms to achieve fine-grained parallelism and asynchronous execution between CPU and GPU tasks. While this highly heterogeneous and deeply overlapping execution pattern significantly improves throughput, it also greatly increases the complexity of system behavior, making the identification and root cause analysis of performance bottlenecks particularly difficult. Therefore, an efficient and accurate performance profiling and diagnostic method is needed for such heterogeneous parallel training scenarios.

[0004] Traditional performance analysis tools such as Perf [1] NVIDIA Nsight Systems [2] Traditional time-accumulation models typically rely on statistical analysis of the CPU or GPU time consumed by each function or code segment during execution to identify the longest-running "hotspots" as potential performance bottlenecks. However, their core assumption—that "the function with the longest execution time is the bottleneck"—often fails in modern deep learning training scenarios. Training large models involves numerous asynchronous operations, pipelined parallelism, and inter-device communication. Key performance bottlenecks may not manifest as localized high execution times, but rather as synchronization dependencies or resource contention across devices and threads. These structural problems cannot be revealed by simple time accumulation.

[0005] Recent work: HTA (Holistic Trace Analysis) [3]The concept of Critical Path Analysis (HTA) is introduced, modeling the program execution process as a Directed Acyclic Graph (DAG). End-to-end performance bottleneck analysis is then performed based on events along the critical paths in the graph. While HTA can more accurately identify the hotspots that truly impact overall performance, it still has significant limitations in scenarios involving large model training, primarily in the following three aspects:

[0006] I. Inadequate Cross-Thread Dependency Modeling: In mainstream deep learning frameworks (such as PyTorch), the main thread is responsible for forward computation and task scheduling, while the automatic differentiation engine creates independent child threads to perform backpropagation. There is a strict synchronization and data dependency relationship between the two. Existing methods assume that CPU threads are independent of each other, failing to accurately model the implicit dependency between the main thread and the automatic differentiation thread. This results in the dependency graph lacking key connections, thus affecting the completeness and accuracy of critical path identification.

[0007] Second, it only supports single critical path identification: In large model training, CPUs and GPUs, as well as different CUDA streams, often execute multiple events in parallel. When backtracking to identify critical paths, the current node may have multiple predecessor nodes with the same end time, thus forming multiple critical paths. Existing methods only identify the single longest critical path and cannot discover hotspots on other paths of equal length, resulting in incomplete hotspot analysis results.

[0008] Third, the lack of a dynamic optimization guidance mechanism: In actual optimization, when developers optimize certain hotspots, the computational dependencies of the program will change accordingly, and new critical paths and hotspots may emerge. Existing methods use static analysis, which cannot continuously track the dynamic evolution of critical paths, lacks systematic guidance for the performance optimization iteration process, and does not define criteria for determining the optimization endpoint.

[0009] Therefore, there is an urgent need for a hotspot identification method that combines cross-thread dependency analysis and multi-critical path identification mechanisms to improve the accuracy of critical path modeling and the comprehensiveness of hotspot assessment, and to provide iterative performance improvement guidance for model training optimization. Summary of the Invention

[0010] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method, system, and application for identifying hotspots in model training that combines cross-thread dependency and multi-critical-path analysis.

[0011] This invention is mainly used to solve the following key technical problems:

[0012] (1) Solve the problem of hot spot misjudgment caused by computational overlap in traditional time accumulation method: By constructing the event dependency graph of program execution and critical path analysis, distinguish between hot events that truly affect end-to-end performance and "false hot events" that are masked by parallel computing, and avoid incorrect allocation of optimization resources.

[0013] (2) Solve the problem that existing critical path analysis tools cannot accurately model cross-thread dependencies: Design an effective cross-thread dependency identification and modeling mechanism to accurately capture complex thread collaboration patterns such as the main thread and automatic micro-thread in PyTorch training, and ensure the integrity and accuracy of critical path construction.

[0014] (3) Solve the problem of ignoring parallel critical paths in single path analysis: Develop a multi-critical path identification algorithm to identify all equal-length critical paths that may affect the overall performance of the program execution, and ensure the comprehensiveness and completeness of hotspot identification.

[0015] (4) Provides an efficient, accurate and non-intrusive automated hotspot identification method: Based on the program runtime performance tracking data collected by existing performance tracking tools, it realizes automated computation dependency graph construction, critical path identification and hotspot ranking, providing scientific and accurate guidance for large model training performance optimization.

[0016] (5) Design a guiding algorithm for dynamic iterative performance optimization: Design a theoretical method that can provide structured guidance for the dynamic iterative performance optimization process. After each round of optimization, new hot spots are re-identified based on the changes in the critical path, and the concept of "optimization endpoint" is scientifically defined to avoid premature termination or blind continuation of the optimization process, and to provide a systematic iterative guidance scheme for continuous performance improvement of large model training.

[0017] Specifically, the present invention is achieved through the following technical means:

[0018] A method for identifying hotspots in model training that combines cross-thread dependency and multi-critical-path analysis, the method comprising:

[0019] Step 1: Collect the event execution trajectory during the model training process, and parse and standardize the collected raw performance data into a unified format;

[0020] Step 2: Aggregate the original events through hierarchical modeling and construct edge connections based on dependencies to form a computational dependency graph;

[0021] Step 3: Identify the sequence of decisive events and the critical paths in the model training process based on the constructed computational dependency graph;

[0022] Step 4: Through overlap analysis and statistical processing, extract hot events with practical optimization value from the critical path and generate a sorted list.

[0023] Step four may also include: providing optimization guidance to developers through dynamic algorithms based on computational dependency graphs and critical path information, and scientifically defining the criteria for determining the optimization endpoint.

[0024] In the specific implementation of this invention, an identification strategy based on event timestamps and actual dependencies is used to identify single critical paths and / or multiple critical paths.

[0025] And / or,

[0026] In the process of single critical path identification, an endpoint backtracking strategy is adopted. The current node is initialized as the last node to end in the computation dependency graph. Then, in the loop, the path is dynamically selected according to the dependency type between events and the predecessor node with the latest end time is backtracked until the starting point is reached.

[0027] In the process of identifying multiple critical paths, all predecessor nodes with the same end time are recursively explored, independent path copies are generated, and the operation is repeated to explore and identify all possible paths.

[0028] The present invention also provides a system for implementing the above method, comprising: a performance tracing data acquisition module, a computation dependency graph construction module, a critical path identification module, a hotspot list generation module, and an iterative performance optimization guidance module;

[0029] The performance tracking data acquisition module collects the event execution trajectory during the training process of the large model and parses the raw performance data into a unified format.

[0030] The computation dependency graph construction module receives the parsed performance data, adds cross-thread dependency modeling, and constructs an event dependency graph.

[0031] The critical path identification module, based on the constructed computational dependency graph, enables parallel identification of multiple critical paths;

[0032] The hotspot list generation module generates a ranking of hotspots based on the critical path analysis results;

[0033] The iterative performance optimization guidance module dynamically sets optimizations based on the computation dependency graph and critical path information.

[0034] The present invention also provides the application of the above-mentioned method or system in characterizing and identifying hotspots in large model training loads and optimizing large model training performance.

[0035] The beneficial effects of this invention include: This invention proposes a model training hotspot identification method and system that combines cross-thread dependency and multi-critical-path analysis, comprising five modules: data preprocessing, computational dependency graph construction, critical path identification, hotspot list generation, and iterative performance optimization guidance. The data preprocessing module standardizes runtime performance tracing data files generated by different performance analysis tools, constructing a unified event data model; the computational dependency graph construction module uses the preprocessed event data to construct a computational dependency graph reflecting the CPU-GPU heterogeneous computing dependencies; the critical path identification module identifies critical paths based on the constructed computational dependency graph, handling both single and multiple critical path scenarios simultaneously; the hotspot list generation module extracts hotspot events with practical optimization value from critical paths and generates a sorted list through precise overlap analysis and statistical processing; and the iterative performance optimization guidance module provides developers with systematic optimization path planning and scientific optimization termination determination through dynamic algorithms, avoiding resource waste caused by blind optimization.

[0036] This invention identifies hotspots in large model training performance based on critical path analysis theory. It proposes a complete heterogeneous computing dependency modeling scheme that accurately handles complex dependencies between CPUs and GPUs, avoiding the "false hotspot" problem caused by traditional methods ignoring heterogeneous characteristics. It also solves the problems of incomplete and erroneous critical path results in existing methods. In comparison, this invention can obtain more comprehensive performance bottleneck information of CPU-GPU heterogeneous computing during training, achieving more accurate and effective hotspot identification, and further improving the targeting and effectiveness of performance optimization. Furthermore, the data preprocessing, dependency graph construction, multi-critical path identification, and hotspot list generation modules of this invention have good versatility and scalability. Therefore, this invention can be applied to large model training scenarios on different deep learning frameworks and hardware platforms, providing accurate performance analysis support for various training environments.

[0037] For AI training platform providers, to ensure efficient execution of large model training tasks and improve user experience, many are working on performance optimization based on the PyTorch deep learning framework. This invention can help platform providers accurately identify hotspots in large model training under different hardware configurations and framework combinations. For deep learning developers, this invention provides a systematic hotspot identification method for complex large model training scenarios, compensating for the shortcomings of existing tools in hotspot analysis for large model training. For performance optimization researchers, traditional performance analysis methods are often based on simple time statistics, which are difficult to reveal the true bottlenecks of CPU-GPU heterogeneous computing in deep learning training. Therefore, hotspot detection methods based on computational dependency graph construction and critical path identification can provide more scientific analytical tools and theoretical support for performance optimization research.

[0038] This invention utilizes performance analysis and load characterization for large-scale model training. The entire hotspot detection process is automatically completed based on runtime performance tracking data collected during training, avoiding the problems of manual judgment and multiple trials required by traditional methods. Simultaneously, event data preprocessing, dependency graph construction, critical path identification, and hotspot statistical ranking are all automatically completed within this invention, eliminating the need for users to possess in-depth performance analysis expertise or complex manual configurations. This enhances user convenience and lowers the technical threshold for performance optimization. By providing a clear hotspot ranking list and quantified performance impact analysis, this invention offers developers clear optimization directions and priority guidance, significantly improving the efficiency and accuracy of large-scale model training performance optimization. Furthermore, by establishing a scientific optimization termination mechanism, this invention ensures comprehensive hotspot identification while avoiding over-optimization, providing a complete solution for large-scale model training performance optimization. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a conceptual relationship diagram of the present invention.

[0041] Figure 2 This is the system architecture diagram of the present invention.

[0042] Figure 3 This is a schematic diagram of the performance tracking data acquisition process of this invention.

[0043] Figure 4 This is a schematic diagram of the computation dependency graph construction module of the present invention.

[0044] Figure 5 This is a flowchart of the critical path identification algorithm of the present invention.

[0045] Figure 6 This is a flowchart illustrating the critical path identification module of the present invention.

[0046] Figure 7 This is a schematic diagram of the hotspot list generation module processing flow of the present invention.

[0047] Figure 8 This is a schematic diagram of the algorithm flow of the iterative performance optimization guidance module of this invention.

[0048] Figure 9 This is a schematic diagram of the implementation scheme of Embodiment 1 of the present invention.

[0049] Figure 10 This is a schematic diagram of the implementation scheme of Embodiment 2 of the present invention.

[0050] Figure 11 This is a schematic diagram of the implementation scheme of Embodiment 3 of the present invention.

[0051] Figure 12 This is a schematic diagram of the implementation scheme of Embodiment 4 of the present invention. Detailed Implementation

[0052] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0053] This invention proposes a method and system for identifying hotspots in model training that combines cross-thread dependency and multi-critical-path analysis. This method accurately identifies true hotspots (performance bottlenecks) during large model training, effectively avoiding misjudgments caused by computational overlap in traditional methods. It also solves the problems of incomplete and incorrect critical paths caused by inaccurate modeling of cross-thread dependencies and neglecting multiple critical path scenarios in the latest technologies. Furthermore, a dynamic iterative performance optimization guidance algorithm is designed, providing developers with systematic optimization guidance and scientifically determining the optimization endpoint. It should be noted that the "event" in this invention refers to events that can be generated by PyTorch Profiler during training execution. [4] The basic execution units captured and identified individually include, but are not limited to, CPU-side operators, function calls in the Python call stack, GPU kernel, and GPU memory allocation or data transfer operations. The relationships between other concepts involved in this invention are as follows: Figure 1 As shown. Figure 2As shown, this invention divides the model training hotspot identification system, which combines cross-thread dependency and multi-critical-path analysis, into five modules: a performance tracking data acquisition module, a computational dependency graph (CDG) construction module, a critical path identification module, a hotspot list generation module, and an iterative performance optimization guidance module. The performance tracking data acquisition module collects event execution trajectories during large model training using PyTorch Profiler and parses the raw performance data into a unified format. The computational dependency graph construction module receives the parsed performance data and, based on existing methods, adds cross-thread dependency modeling to construct a more complete and accurate event dependency graph. The critical path identification module, based on the constructed computational dependency graph, extends traditional single critical path analysis to support parallel identification of multiple critical paths. The hotspot list generation module generates accurate hotspot rankings based on critical path analysis results. The iterative performance optimization guidance module, based on the computational dependency graph and critical path information, provides developers with systematic optimization guidance and scientifically determines the optimization endpoint through a dynamic iterative algorithm. The first four modules form a complete data processing chain, with the output of one module serving as the input of the next. The final iterative performance optimization guidance module, combining the first four modules, provides an overall optimization framework to ensure the organic unity of the entire system. The specific principles and detailed workflows of each module are described below.

[0054] I. Performance Tracing Data Acquisition Module

[0055] To support performance data acquisition in large-scale model training scenarios, the performance tracking data acquisition module of this invention uses PyTorch Profiler, which can provide detailed CPU and GPU event information, including CPU operators, Python call stack functions, GPU Kernel, GPU memory operations, and other event information. By configuring relevant parameters of PyTorch Profiler, this module ensures that it can capture the complete event sequence during large-scale model training, including detailed information on each stage such as forward computation, backpropagation, and optimizer updates.

[0056] like Figure 3As described above, after the large model training begins, the performance tracking data acquisition module first performs a warm-up period by configuring warm-up parameters, skipping the unstable phase of the initial training stage and ensuring that the acquired data reflects the true performance characteristics under stable conditions. After the warm-up period, PyTorch Profiler is enabled for data acquisition. The acquisition configuration includes enabling CPU event tracing, CUDA event tracing, and Python call stack (optional). During the acquisition process, CUDA synchronization event capture needs to be additionally enabled by setting PyTorch Profiler parameters to ensure accurate identification of synchronization dependencies between CUDA Streams, providing the necessary data foundation for subsequent computation dependency graph modeling. After acquisition, the raw PyTorch Profiler data is parsed into a standardized event sequence containing key fields such as event ID, event name, start time, execution time, and execution thread / CUDA Stream, and stored in a JSON file.

[0057] II. Computation Dependency Graph Construction Module

[0058] The computational dependency graph construction module of this invention innovatively introduces a cross-thread dependency modeling mechanism to achieve accurate modeling of complex heterogeneous execution scenarios. Computational dependency graph construction is one of the core steps of this invention, which involves converting collected performance tracing data into a computational dependency graph that accurately reflects the program's execution logic. This step includes two key components: event aggregation and dependency modeling.

[0059] like Figure 4 As shown, the dependency graph construction module takes a standardized event sequence (JSON file) as input and outputs a complete event computation dependency graph containing cross-thread dependencies. This module employs a hierarchical modeling approach. First, it intelligently aggregates the original events to reduce graph structure complexity. Then, it constructs precise edge connections based on various dependency types, ultimately forming a computation dependency graph that accurately reflects the training execution logic of large models. Compared to existing methods, this invention places particular emphasis on the identification and modeling of cross-thread dependencies, ensuring the capture of complete dependencies within the PyTorch multi-threaded training architecture.

[0060] Regarding event aggregation strategies, to reduce the complexity of the computational dependency graph structure and improve the efficiency of subsequent critical path identification, this invention employs a multi-dimensional aggregation strategy to organize the original events into computational dependency graph nodes. This reduces the number of nodes in the constructed dependency graph, lowers the graph's size and complexity, and improves the speed of subsequent critical path identification. The core principle is to ensure that events within the same node after aggregation only have sequential execution dependencies. First, events are divided according to the training phase. Based on the semantics of events in the training process, they are categorized into six main phases: data loading, forward propagation, loss calculation, back propagation, communication, and parameter update. This division not only reduces the number of nodes but also preserves the semantic integrity of the training process. Second, CUDA Stream boundary identification is performed. When a CPU event calls a GPU event and a CUDA Stream switch occurs, the aggregation of the current node needs to end and a new node needs to be started. This is because GPU events on different Streams can be executed in parallel, and there is no sequential execution dependency between them; only synchronization wait constraints are possible. Finally, synchronization operation identification is implemented. Events before and after synchronization operations are specially marked. These events are often key points of CPU-GPU interaction and need to be differentiated and processed during aggregation.

[0061] In terms of dependency modeling, this invention constructs edge connections in the computation dependency graph based on four core dependency relationships: sequential execution dependency, invocation dependency, synchronization dependency, and cross-thread dependency. Its core principle is to accurately reflect the real dependency relationships between nodes during program execution.

[0062] Sequential execution dependencies reflect the fundamental constraints of hardware execution, namely that event nodes within the same CPU thread form a dependency chain according to the execution order, and GPU events on the same CUDA Stream also follow strict sequential dependencies.

[0063] Call dependencies reflect the CPU's control relationship with the GPU. CUDA Kernel or GPU memory operation events depend on the CPU events that initiate the calls. It should be noted that due to the asynchronous execution characteristics of the GPU, this dependency is unidirectional.

[0064] Synchronization dependencies are a relatively complex type of dependency in heterogeneous computing, including two cases: CPU-GPU synchronization dependencies and GPU-GPU synchronization dependencies. The former uses calls such as cudaDeviceSynchronize and cudaStreamSynchronize to make the CPU wait for the GPU to complete, while the latter uses mechanisms such as cudaEventRecord and cudaStreamWaitEvent to implement synchronization constraints between different streams.

[0065] Cross-thread dependency is a key innovation of this invention. The PyTorch automatic differentiation engine employs a multi-threaded architecture for gradient computation, where the main thread handles forward computation and training process control, while the automatic differentiation engine creates dedicated worker threads to perform backpropagation computation. However, due to the existence of the Python Global Interpreter Lock (GIL), multiple Python threads actually execute Python code sequentially and alternately, resulting in implicit execution dependencies between threads. In actual execution, when the main thread calls the `loss.backward()` method, the automatic differentiation engine distributes the backpropagation task to worker threads, and the main thread then enters a waiting state until all gradient computations are complete. Existing dependency graph construction methods assume that different threads are independent of each other, considering only intra-thread event dependencies. This leads to incomplete dependency modeling in large-scale PyTorch model training scenarios. Therefore, based on the Python GIL mechanism, this invention adopts a simple and effective cross-thread dependency handling method: treating the events of the main thread and the automatic differentiation thread as sequentially executed within the same logical thread, and establishing dependencies according to the actual timestamp order of the events. The specific process is as follows:

[0066] The first step is to uniformly sort the event timestamps. This invention collects all events from the main thread and the automatic micro-molecule thread, and uniformly sorts them according to the start timestamp of the events to form a global event execution sequence.

[0067] The second step is to establish dependencies based on the GIL constraints. Since the GIL guarantees the serial execution of Python code, this invention establishes dependencies between time-adjacent events from the same or different threads. That is, for a sorted sequence of events, a later-executed event depends on an earlier-executed event, regardless of whether the two events originate from the same thread.

[0068] The third step is to construct cross-thread dependency edges. Directed edges are added to the computation dependency graph, running from the earlier executed event to the later executed event. These edges cross thread boundaries and accurately reflect the actual execution dependencies under the GIL constraints.

[0069] By employing the unified processing method based on timestamp sorting, this invention accurately identifies cross-thread dependencies and adds corresponding dependency edges to the computational dependency graph, ensuring that the dependency graph fully reflects the actual execution logic of PyTorch training. This approach reflects the actual execution mode under GIL constraints while avoiding complex cross-thread synchronization analysis, ensuring the correctness of dependency graph construction and subsequent critical path identification. Through the aforementioned event aggregation strategy and dependency modeling, this invention ultimately constructs a computational dependency graph that accurately reflects the program execution logic, laying a solid foundation for subsequent critical path identification and hotspot extraction.

[0070] III. Critical Path Identification Module

[0071] The main innovation of the critical path identification module in this invention compared to existing methods lies in its expanded multi-critical path identification capability. Traditional critical path analysis methods only identify the single longest path in the computation dependency graph, ignoring the possibility of multiple critical paths of equal length in heterogeneous parallel computing. In large model training scenarios, when GPU computation and CPU data preprocessing, as well as parallel events between multiple CUDA streams, end at the same time, multiple critical paths will be formed. Optimizing only one path cannot achieve an overall performance improvement.

[0072] The critical path identification module of this invention aims to accurately identify the event sequence that determines the end-to-end execution time of a program from a constructed computational dependency graph. This invention employs an identification strategy based on event timestamps and actual dependencies, rather than a simple longest path algorithm, to avoid misjudgments in complex dependency scenarios. Simultaneously, this module uses a stack-based depth-first search algorithm, combined with path copying and recursive exploration mechanisms, to simultaneously identify all critical paths in the computational dependency graph that meet the conditions.

[0073] The flowchart of the specific critical path identification method is as follows: Figure 5 As shown. In single critical path identification, the method of this invention adopts a backtracking strategy from the endpoint. First, the current node is initialized as the last node to finish in the computation dependency graph, and then its predecessor nodes are continuously backtracked in a loop until the starting point is reached. During the predecessor node selection process, this invention dynamically selects the path based on the dependency type between events: when the current node is located on the CPU, if the node synchronously depends on other nodes, the corresponding predecessor node on the GPU is preferentially selected as the predecessor node on the critical path, because the synchronization semantics indicate that the start time of the CPU node is constrained by the completion time of the GPU node it synchronously depends on; if there is no synchronous dependency, it continues to be included forward along the CPU-side dependency. When the current node is located on the GPU, the effective end time of all predecessor nodes needs to be accurately calculated: for GPU predecessor nodes, their actual end time is used directly; for CPU predecessor nodes, the time of the first cudaLaunchKernel event (the CUDA runtime event that starts the GPU Kernel) is found as the effective end time, and then the predecessor node with the latest effective end time is selected for inclusion in the critical path. Each time a predecessor node is identified on the critical path, it is added to the critical path and updated as the new current node. This process is repeated until the starting point is reached, and finally a complete critical path is constructed.

[0074] The extended multi-critical-path identification is a key innovation of this invention, overcoming the limitations of traditional single-critical-path analysis. During GPU node backtracking, when encountering multiple predecessor nodes with the same maximum effective end time, the algorithm no longer selects only one path but recursively explores all predecessor nodes with equal times, ensuring no possible critical paths are overlooked. Specifically, the algorithm first identifies all predecessor nodes with an end time equal to the maximum effective end time, then creates an independent exploration path for each isochronous predecessor: by copying the current path state, an independent path copy is generated for each predecessor node; when the predecessor is a CPU node, overlapping events masked by the GPU are filtered out; the processed predecessor node is appended to the corresponding path and pushed onto the exploration stack. Through recursive processing via the stack mechanism, when an isochronous predecessor node encounters multiple isochronous predecessors again in subsequent backtracking, the algorithm continues to apply the same analysis strategy, continuously branching and exploring all possible path combinations until all critical paths are fully identified.

[0075] like Figure 6 The critical path identification module of this invention uses a computational dependency graph as input. This module first determines the endpoint event of the entire program and begins backtracking. During backtracking, if only one critical predecessor exists in each step, a single-path identification strategy of "backtracking to the starting point" is adopted until a complete critical path is constructed. When multiple latest predecessor nodes with the same end time are encountered, it indicates the existence of "multiple" critical paths. At this point, the algorithm switches to a "recursive backtracking" strategy, creating and exploring branch paths of all isochronous predecessors to ensure that all possible critical paths are fully identified. Ultimately, this module can accurately output all critical paths that determine the end-to-end execution time of the program, laying a solid foundation for subsequent hotspot analysis.

[0076] IV. Hotspot List Generation Module

[0077] The hotspot list generation module generates an accurate hotspot ranking based on critical path identification results, through overlapping event filtering and statistical aggregation. Unlike traditional hotspot identification methods based on global time accumulation, this invention's hotspot evaluation specifically analyzes events within the critical path and avoids time duplication issues caused by heterogeneous computing overlap by excluding overlapping portions resulting from parallel execution between CPU nodes and GPU nodes with call dependencies. Hotspot extraction and ranking are crucial steps in transforming critical path analysis results into actionable optimization guidance. Through precise overlap analysis and statistical processing, hotspot events with practical optimization value are extracted from the critical path.

[0078] The execution flow of the hotspot list generation module is as follows: Figure 7As shown, this module receives critical path information output by the critical path identification module and generates a hotspot list. Before hotspot extraction, it is necessary to analyze the time overlap between CPU nodes and their called GPU nodes in the critical path. When both a CPU node and its called GPU node are on the critical path, their actual time contributions need to be carefully calculated to avoid duplicate statistics. The specific processing strategy is to identify the position of the first cudaLaunchKernel event in the CPU node, and only count events before its end time as valid events for that CPU node, thereby excluding the part that overlaps with GPU execution. For the excluded overlapping events, if they are not in other critical path segments, they are not included in the hotspot statistics, ensuring that the time contribution of each event is accurately calculated. At the same time, in scenarios with multiple critical paths, it is necessary to identify global hotspots through path intersection analysis. According to critical path theory, only by shortening the length of all critical paths simultaneously can the execution time of the entire program be shortened. Therefore, the intersection of all critical paths is the real bottleneck affecting the overall performance of the program. This invention provides a more comprehensive analytical basis for hotspot evaluation by identifying the intersection points and common path segments between multiple critical paths.

[0079] In the hotspot statistics and sorting phase, this invention performs a systematic aggregation analysis on all events retained on the critical path. First, events are grouped by name, with identical names merged into the same group. Then, the total effective execution time for each group is calculated. Next, the proportion of each group's events in the total critical path time is calculated; this metric directly reflects the impact of this type of event on overall performance. Finally, events are sorted in descending order of total execution time, and a final hotspot list is generated based on the number of hotspots extracted according to the user's settings. This list clearly displays the hotspot events that truly affect training performance, providing developers with clear optimization directions and priority guidance, ensuring that optimization efforts focus on the key bottlenecks that truly impact the training performance of large models.

[0080] V. Iterative Performance Optimization Guidance Module

[0081] The iterative performance optimization guidance module of this invention is a significant extension of critical path construction and hotspot identification. It aims to provide structured dynamic performance optimization guidance and scientific endpoint determination for large model training through an iterative performance optimization guidance strategy based on critical path identification. This module addresses the lack of a systematic iterative performance optimization guidance framework and a lack of scientific, data-driven endpoint determination criteria in existing performance analysis tools. By establishing a clear concept of "optimization endpoint" and an iterative guidance mechanism, this module effectively avoids the underutilization of performance potential due to premature termination of the performance optimization process and the waste of resources caused by blindly continuing performance optimization work.

[0082] In actual performance optimization, a complete performance optimization cycle includes three core stages: performance tracing data acquisition (measurement), computation dependency graph construction + critical path identification + hotspot list generation (analysis), and execution of specific optimization operations based on the hotspot list (optimization). When a developer completes optimization operations on a hotspot event, the execution characteristics of the computation graph change; the original critical path may become invalid, and new hotspots may emerge. For example, after optimizing a GPU computation hotspot, a previously masked CPU data preprocessing operation may be on a new critical path and become a new hotspot; or in a multi-critical-path scenario, when the hotspots in the common parts of all critical paths are optimized, the original multi-critical-path situation may transform into a single-critical-path scenario due to the elimination of common bottlenecks, forming a new critical-path structure. Existing static analysis methods cannot provide continuous guidance for such dynamic changes, often leaving developers in a dilemma of "not knowing what to optimize next" and "not knowing when to stop performance optimization." This module provides developers with a guide to the next steps after each round of optimization by establishing an iterative guidance framework based on dynamic identification of critical paths and event handling status management. Through a scientific termination condition determination mechanism, it clearly defines a reasonable endpoint for performance optimization work, ensuring the systematic, complete and efficient nature of performance optimization work.

[0083] To effectively manage the iterative performance optimization process, this module introduces an event processing status marking mechanism. This mechanism has two core purposes: first, it records whether hot events identified in the current iteration have already been identified as optimization targets and processed in previous iterations, thus avoiding the repeated processing of the same optimization targets in subsequent iterations and ensuring the efficiency and systematic nature of the iterative process; second, when all events on the critical path identified in a given iteration are marked as processed, it indicates that the program has no further room for optimization, and the iteration ends, thus providing a scientific basis for determining optimization termination. Each event maintains a "processing status" identifier, including "unprocessed" and "processed" states: "unprocessed" indicates that the event has not yet been identified as an event on the critical path requiring optimization consideration in any iteration; "processed" indicates that the event has been identified as being on the critical path in an iteration and has been considered for optimization, including generating corresponding optimization guidance as a hot event, and cases where it was not included in the hot event list due to user-set hot event quantity thresholds or was deemed not worth optimizing. It is important to emphasize that the actual performance optimization operations (such as algorithm improvement, parameter tuning, kernel optimization, etc.) are performed by developers or automatic tuning systems after the hotspot list is generated. This algorithm is mainly responsible for providing optimization direction and iteration control, rather than executing specific optimization implementations. The dynamic management of event processing status follows these principles: when an event is identified as being on the critical path in the current iteration, it will be marked as "processed," and the user will be expected to optimize it to the maximum extent possible. In subsequent iterations, events marked as "processed" will no longer be considered new optimization targets; the algorithm will focus on events with a "not processed" status. Actual optimization operations may cause changes in the structure of the dependency graph or execution time. Therefore, in the next iteration, the algorithm will reconstruct the dependency graph, identify critical paths, and update processing status based on newly collected performance tracking data, thus forming a continuously iterative performance optimization guidance closed loop.

[0084] like Figure 8As shown, the core algorithm flow of the iterative performance optimization guidance module mainly includes key steps such as performance tracking data collection, computational dependency graph construction, critical path identification, hotspot list generation, user optimization, event handling status marking strategy, and iteration termination condition judgment. The algorithm first initiates the iterative process based on a large model training load and sets corresponding auxiliary iteration termination conditions according to user settings. In each iteration, the algorithm executes a complete optimization loop sequentially: first, it re-collects performance data under the current state through the performance tracking data collection module (measurement); then, it analyzes the performance data through the computational dependency graph construction, critical path identification, and hotspot list generation modules (analysis); finally, it guides developers or automatic tuning systems to perform specific optimization operations based on the generated hotspot list (optimization). In the analysis phase, the algorithm identifies the current critical paths and determines their number, then adopts different processing strategies based on the number of critical paths: when there is only a single critical path, it focuses on all events on that critical path; when there are multiple critical paths, it focuses on events on the intersection of multiple critical paths. The algorithm avoids repeatedly processing the same optimization objectives by using an event processing status marking mechanism: if all events within the region of interest are marked as "processed," it means that all events on the current critical path have been considered and processed in previous iterations, indicating that there is no more room for optimization, and the performance optimization iteration process ends at this point; otherwise, events in the region of interest that are currently "unprocessed" are marked as "processed," indicating that these events will be processed in the current iteration's optimization operations, and a hotspot list is generated for developers to optimize. Simultaneously, after each iteration, the algorithm checks whether preset auxiliary termination conditions (such as time limits, round limits, etc.) are met. If they are met, the performance optimization iteration process ends early; otherwise, it proceeds to the next iteration. This iterative mechanism ensures the dynamism and continuity of performance optimization guidance, providing developers with a complete guidance path for performance optimization iterations from start to finish.

[0085] Specifically, in handling single-critical-path scenarios, when the system identifies a single critical path, it focuses on all events on that critical path. First, it checks the event processing status on the critical path. If all events are marked as "processed," it indicates there are no new optimization targets on the current critical path, and the performance optimization iteration ends. Otherwise, the algorithm uses a full-path event marking strategy, marking all events on the current critical path as "processed," indicating that these events will be processed in this iteration. Then, a hotspot list is generated for developers to optimize. In handling multi-critical-path scenarios, when the system identifies multiple critical paths, it focuses on events at the intersection of these paths. First, the intersection of multiple critical paths is calculated, identifying event nodes shared by all paths. The processing status of these intersection events is checked. If all intersection events are marked as "processed," it indicates that the common bottlenecks affecting all critical paths have been addressed, and the performance optimization iteration ends. Otherwise, the algorithm uses a path intersection event marking strategy, marking all intersection events as "processed," indicating that these events will be processed in this iteration. A hotspot list is then generated for developers to optimize. In the next iteration, when re-identifying critical paths, events marked as "processed" are considered to have been processed in previous iterations and are no longer considered for optimization. Only unmarked new hotspots need to be considered. Furthermore, in the next iteration, the critical path structure or number may change. The algorithm will determine the number of paths and select an appropriate processing strategy for iterative optimization.

[0086] To adapt to optimization scenarios of varying scales and complexities, this invention designs a flexible iteration termination mechanism. The primary termination condition is that when all events within the region of interest on the critical path in a given iteration have been marked as "processed," it indicates that no new events can be optimized in the current state, and the iteration process automatically stops. Furthermore, the algorithm supports several user-defined auxiliary termination conditions: a time constraint strategy allows users to set an upper limit on the total optimization time, automatically stopping iteration when the preset time is reached; a round limit strategy allows users to set a maximum number of iteration rounds to avoid excessive iterations; and an optimization event quantity strategy allows users to set a maximum number of optimization events, stopping iteration when the total number of optimized events reaches a preset value. After each iteration, the system checks whether any of the auxiliary termination conditions are met; if so, the entire performance optimization process ends prematurely.

[0087] In the following embodiments, Model_A, Model_B, and Model_C represent large models of different scales. Model_A represents a medium-sized model with 1B-10B parameters, such as GPT-2 and BERT-Large; Model_B represents a large model with 10B-100B parameters, such as GPT-3 and Llama2-70B; and Model_C represents a very large model with more than 100B parameters, such as GPT-4 and PaLM. Training_X, Training_Y, and Training_Z represent different training modes, corresponding to single-GPU training, multi-GPU data-parallel training, and large-scale distributed training, respectively. System_1, System_2, and System_3 represent the computing environments based on the above training modes. In the distributed training scenario, System_1, System_2, and System_3 represent training systems with the same hardware configuration and the same software environment, and all of them perform the same type of training task on the same large model.

[0088] Example 1: Load Characterization and Hotspot Identification

[0089] This embodiment illustrates the process by which the present invention characterizes load features and identifies critical paths and hotspots during model training. In this embodiment, a large model of Model_A size is trained in the System_1 computing environment of Training_X mode. This embodiment aims to complete the construction of the computation dependency graph, critical path identification, and accurate analysis of hotspot events to achieve accurate load characterization.

[0090] The execution flow of Example 1 is as follows: Figure 9 As shown, the training load of Model_A in Training_X mode is first started in System1. Then, the performance tracing data acquisition module begins to collect and parse PyTorch runtime performance data within a specified interval of the training process. During the data acquisition phase, this invention specifically enables the capture function of CUDA synchronization events to ensure that implicit dependencies between CUDA streams can be identified. During the acquisition process, the performance tracing data acquisition module uses PyTorch Profiler technology to collect events such as CPU operators, Python call stack functions, GPU Kernel, and GPU memory operations. The collected event data includes key fields such as event ID, event name, start time, end time, and execution thread identifier, and is parsed into JSON format program performance tracing data.

[0091] The computation dependency graph construction module then takes performance tracking data as input and outputs a complete computation dependency graph based on event aggregation strategies and dependency modeling. During event aggregation, the algorithm semantically partitions events according to six training phases: data loading, forward propagation, loss calculation, back propagation, communication, and parameter update. When a CUDA Stream switch is detected, the algorithm immediately ends the aggregation of the current node and creates a new node, as GPU events on different Streams can be executed in parallel. Simultaneously, the algorithm specially marks events before and after synchronization operations such as `cudaDeviceSynchronize` and `cudaStreamSynchronize` to ensure that synchronization constraints are accurately reflected in the dependency graph. In the dependency modeling phase, the algorithm constructs edge connections based on four core dependency relationships: sequential execution dependencies ensure the order constraints of events within the same CPU thread or CUDA Stream; call dependencies reflect the CPU's control relationship over the GPU; synchronization dependencies handle complex constraints of CPU-GPU synchronization and GPU-GPU synchronization; and cross-thread dependencies establish time-series dependencies between the main thread and the automatic differentiation thread based on the GIL mechanism. During cross-thread dependency construction, the algorithm uniformly sorts all events of the main thread and the automatic differentiation thread according to their start timestamps, forming a global event execution sequence. When the main thread calls the `loss.backward()` method (timestamp T1), the automatic differentiation engine begins gradient calculation in worker threads (timestamp T2, T2 > T1). At this point, the algorithm establishes cross-thread dependency edges in the computation dependency graph, from the backpropagation call event of the main thread to the gradient calculation event of the automatic differentiation thread. Similarly, when the automatic differentiation thread completes gradient accumulation (timestamp T3), the main thread continues to perform parameter update operations (timestamp T4, T4 > T3), and the algorithm establishes corresponding cross-thread dependency edges.

[0092] Next, the critical path identification module uses an algorithm that backtracks from the endpoint to identify all critical paths. The algorithm starts backtracking analysis from the node that last ends in the computation dependency graph. During backtracking, for the selection of predecessors for GPU nodes, the algorithm calculates the effective end time of all predecessor nodes: for GPU predecessors, its actual end time is used directly; for CPU predecessors, the time of the first cudaLaunchKernel event is used as the effective end time. When multiple predecessor nodes are found to have the same maximum effective end time, the algorithm initiates a multi-critical path identification mechanism. At this point, instead of selecting a single path, an independent exploration branch is created for each isochronous predecessor node. Path replication is implemented through a stack mechanism: the current path state is copied into multiple copies, each copy corresponding to an isochronous predecessor node, and then all path copies are pushed onto the exploration stack. The algorithm continues to recursively process the paths in the stack until all possible critical paths have been fully identified.

[0093] Finally, the hotspot list generation module generates an accurate hotspot ranking list based on event statistics on the critical path through overlap analysis. During the overlap analysis phase, the algorithm identifies the time overlap relationship between CPU nodes and the GPU nodes they call. When both a CPU node (e.g., linear layer computation) and its called GPU node (e.g., CUDA matrix multiplication kernel function) are located on the critical path, the algorithm finds the position of the first cudaLaunchKernel event in the CPU node, only counting the CPU computation time before that position as the effective contribution, avoiding duplicate statistics with GPU execution time. In multi-critical-path scenarios, the algorithm identifies common events across all critical paths through path intersection analysis. Only events appearing simultaneously on all critical paths are considered true global bottlenecks, because according to critical path theory, only by shortening the length of all critical paths can the overall execution time be reduced. The algorithm sorts the intersection events in descending order of total effective execution time, generating the final hotspot list, achieving the goal of accurately locating training hotspots in Model_A on System_1 and in Training_X training mode, thus completing the load characterization task.

[0094] This embodiment can comprehensively characterize the execution features and performance bottleneck distribution of the model training load, providing baseline data for performance evaluation under different training modes and scales.

[0095] Example 2: Comparison of load characteristics under different implementation schemes

[0096] This embodiment illustrates the implementation process of the present invention for quantitatively comparing and analyzing the model training load characteristics under different implementation schemes. In this embodiment, a large model of Model_B size is trained in the System_2 computing environment of Training_Y mode. This embodiment aims to identify and compare the differences in load characteristics between different schemes based on computational dependency graphs and critical path analysis.

[0097] The execution flow of Example 2 is as follows: Figure 10 As shown, the training task of Model_B in Training_Y mode is first started on System_2. In order to evaluate the performance differences between different implementation schemes, two schemes are selected for the same type of training (e.g., changing operator implementation, using different dependency packages, or adjusting communication strategies). The performance tracking data acquisition module collects and parses PyTorch runtime performance data at the specified intervals during the training process of each scheme.

[0098] During the data collection process, the performance tracing data acquisition module collects events such as CPU operators, Python call stack functions, GPU Kernel, and GPU memory operations during the training process of Scheme A, and parses them into JSON format program performance tracing data. The computation dependency graph construction module constructs a complete computation dependency graph based on the performance tracing data of Scheme A, using an event aggregation strategy and four core dependency relationships. Specifically, through the cross-thread dependency modeling technique detailed in Example 1, the system establishes the complete dependency relationship between the main thread and the automatic differentiation thread in Scheme A. The critical path identification module uses a multi-critical path identification algorithm to backtrack from the endpoint of the dependency graph and identify all critical paths in Scheme A. The hotspot list generation module generates a ranked list of hotspots for Scheme A through precise overlap analysis.

[0099] Similarly, the exact same analysis process was performed on Solution B: performance data was collected, cross-thread dependencies were established, a computation dependency graph was constructed, multiple critical paths were identified, and a hotspot list was generated. The entire analysis adopted the same technical workflow as Solution A to ensure the comparability of the results.

[0100] After the data collection is completed, the hotspot lists generated by Scheme A and Scheme B are quantitatively compared and analyzed. By comparing the total critical path duration and hotspot distribution of each implementation scheme, this invention can quantify the impact of different implementation schemes on overall performance. The comparison includes: (1) Comparison of total critical path duration: Analyze the difference in critical path execution time between Scheme A and Scheme B to quantify the overall performance gap. When the critical path of a certain scheme is significantly shortened, it indicates that the scheme has a greater advantage in execution efficiency. (2) Comparison of hotspot event ranking: Compare the time proportion and ranking changes of each event in the hotspot list of the two schemes to identify the optimization effect of different implementation schemes on specific computational operations. (3) Analysis of hotspot distribution differences: Evaluate the differences in performance bottleneck characteristics of different schemes through indicators such as the concentration of the training stage to which the hotspot belongs and the distribution of hotspot types. A decrease in the time proportion of a certain hotspot event indicates that the corresponding algorithm optimization has achieved results.

[0101] This embodiment can accurately compare the differences in performance structure among various implementation schemes within a consistent analysis framework, providing quantitative basis for performance decisions and optimization directions of large model training systems.

[0102] This embodiment can accurately compare the differences in performance structure among various implementation schemes within a consistent analysis framework, providing quantitative basis for performance decisions and optimization directions of large model training systems.

[0103] Example 3: Analysis of Load Characteristic Changes Before and After Optimization

[0104] This embodiment illustrates the process of systematically analyzing changes in training load structure and hotspot distribution before and after performance optimization according to the present invention. In this embodiment, a large model of Model_B size is trained in the System_2 computing environment of Training_Y mode. This embodiment aims to quantitatively verify the performance optimization effect and perform bottleneck migration analysis by comparing the total critical path duration and hotspot list before and after optimization.

[0105] The execution flow of Example 3 is as follows: Figure 11 As shown, the Model_C baseline training task in Training_Z mode is first run in System_3, and baseline performance data is collected using the performance tracing data acquisition module. The collection scope includes events such as CPU operators, Python call stack functions, GPU Kernel, and GPU memory operations, and is parsed into program performance tracing data in JSON format.

[0106] The complete Implementation 1 process is executed on the baseline configuration before optimization: the dependency graph construction module constructs the baseline dependency graph based on the collected results and establishes the dependency relationship between the main thread and the automatic differentiation thread through cross-thread dependency modeling technology; the critical path identification module uses a multi-critical path identification algorithm to identify the set of critical paths with the longest time consumption; and the hotspot list generation module generates an initial hotspot ranking based on the time proportion of events in the critical path, and determines the hotspot events and the corresponding main performance bottleneck locations (such as communication delay, synchronization wait, data loading, etc.).

[0107] Implementation of performance optimization operations: Next, performance optimization operations are performed on the identified bottlenecks, including but not limited to algorithm reconstruction, operator kernel fusion, communication topology adjustment, or improvement of asynchronous scheduling schemes. Optimization operations target the main hotspots identified in the baseline analysis, employing corresponding optimization strategies for improvement.

[0108] Complete implementation of post-optimization analysis: After optimization, the training task is re-executed under the same conditions to collect new performance data. The complete implementation process of Example 1 is executed again on the optimized configuration: cross-thread dependencies are re-established (which may change due to optimization), the optimized computation dependency graph is constructed, the critical path identification module identifies a new set of critical paths again, and the hotspot list generation module outputs a new hotspot ranking result.

[0109] Comparative analysis of hotspot results before and after optimization: By comparing the hotspot lists before and after optimization, the effectiveness of optimization measures can be quantitatively evaluated: (1) Changes in critical path duration: Compare the total execution time of the critical path before and after optimization and calculate the percentage improvement in performance. A significant reduction in the total duration of the critical path indicates that the performance optimization has achieved positive results. (2) Analysis of changes in hotspot ranking: Compare the changes in the ranking and time proportion of each event in the hotspot list before and after optimization to verify the effect of optimizing bottleneck events. (3) Bottleneck migration identification: Analyze the situation where the concentration of hotspot events decreases or the location of hotspots shifts to reveal the bottleneck migration pattern caused by optimization operations. Significant improvement of the original main hotspots is often accompanied by the emergence of new hotspots, pointing the way for subsequent optimization.

[0110] This embodiment demonstrates that the present invention can not only locate performance bottlenecks, but also verify the effectiveness of optimization measures in a structured manner, providing quantitative basis for performance improvement and guiding subsequent iterative optimization.

[0111] Example 4: Dynamic Hotspot Identification in Iterative Performance Optimization Scenarios

[0112] This embodiment illustrates the process by which the present invention guides continuous performance optimization during model training based on dynamic analysis and feedback mechanisms. In this embodiment, a large model of Model_C size is trained in the System_3 computing environment of Training_Z mode. This embodiment aims to construct a closed-loop iterative system for performance optimization—measurement-analysis-optimization—by introducing event handling state marking and optimization endpoint determination mechanisms. Through this system, dynamic hotspot tracking, optimization effect evaluation, and scientific optimization endpoint determination are completed, thereby achieving continuous improvement in model training performance.

[0113] The execution flow of Example 4 is as follows: Figure 12 As shown, first, the initial training load of Model_C in Training_Z mode is started on System_3 to establish a performance baseline. Then, the iterative performance optimization guidance module enters multiple optimization loops.

[0114] In each iteration, the complete analysis process of Example 1 is executed: the performance tracing data acquisition module collects performance data under the current optimization state; the dependency graph construction module regenerates the dependency graph based on the latest performance characteristics, paying particular attention to the impact of optimization operations on cross-thread dependency structures; the critical path identification module analyzes the changes in the path structure after optimization, using a multi-critical path identification algorithm to ensure that all paths of equal length are discovered; and the hotspot list generation module regenerates a sorted list of hot events based on the critical paths identified in this round. The algorithm performs a critical path number determination in each iteration: when a single critical path is identified, all events on that path are considered; when multiple critical paths are found, the intersection of the paths is the primary focus. This differentiated processing strategy ensures that the optimization work always focuses on the bottleneck events that have the greatest impact on overall performance.

[0115] During iterative optimization, this invention introduces an event processing status marking mechanism to manage optimization progress. Each event maintains two status identifiers: "Unprocessed" and "Processed." "Unprocessed" indicates that the event has not yet been identified as an optimization target on the critical path in any iteration round; "Processed" indicates that the event has been identified as being on the critical path and has been included in the optimization consideration. The algorithm employs persistent state management based on event identifiers: a unique identifier is assigned to each event, and the state record is maintained across iteration rounds. When an event is identified as being on the critical path in the current iteration, the user is prompted to decide whether to optimize it. Regardless of whether the user ultimately chooses to optimize it, it will be marked as "Processed," ensuring that the same optimization target is not considered repeatedly in subsequent iterations.

[0116] When all events on the critical path in this round are marked as "processed," it indicates that the program has no room for further optimization. At this point, the critical path is considered to have converged, and the iterative performance optimization ends. The algorithm implements a multi-level termination strategy: the main termination condition is based on the integrity check of the event processing status; auxiliary termination conditions include user-defined time limits, iteration round limits, and thresholds for the number of optimized events. If there are still unprocessed hot events, the algorithm marks the relevant events on the critical path as "processed" and provides the user with the optimization direction and priority reference for the next round based on the updated hot event list. The user can perform targeted optimization operations based on this guidance information. At the same time, the algorithm makes termination decisions based on user-defined auxiliary termination conditions: when any condition is met, the iteration process ends and a complete analysis report is output; otherwise, the algorithm enters the next iteration and re-executes the complete analysis process of Example 1.

[0117] Through multiple rounds of iterative execution, the guidance algorithm of this invention evolves from static performance analysis to dynamic closed-loop analysis and decision support, continuously providing optimization direction guidance based on real-time performance feedback during training. The algorithm adaptively updates the performance profile and tracks bottleneck evolution in multiple rounds of analysis, automatically determining the analysis endpoint when the critical path converges or a preset termination condition is met. This forms a performance hotspot identification and guidance algorithm with continuous guidance capabilities and a clear optimization endpoint determination mechanism. This embodiment demonstrates the iterative performance analysis and guidance capabilities of this invention in complex training environments and proposes a scientific determination mechanism for optimization endpoints, providing a dynamic and scalable performance optimization guidance framework for large model training processes.

[0118] Through the four embodiments described above, this invention verifies the applicability and effectiveness of its model training hotspot identification method and system, which combines cross-thread dependency and multi-critical path analysis, under different training environments with large-scale models and various modes. This invention can accurately characterize load features and identify hotspots during model training, further accurately locate performance bottlenecks in static and dynamic scenarios, quantitatively compare the differences between different implementation schemes, evaluate the effects before and after optimization, and scientifically determine the optimization endpoint through continuous analysis. Therefore, it provides a dynamic, scalable, and continuously guiding performance analysis and optimization framework for large-scale model training, offering a complete performance optimization technical solution for various large-scale model training scenarios.

[0119] References

[0120] [1]Linux Foundation. perf: Linux performance analysis tool[EB / OL].Linux Foundation, [2025-10-31]. https: / / perf.wiki.kernel.org / .

[0121] [2]NVIDIA Corporation. Nsight Systems User Guide[EB / OL]. Version2025.5.: NVIDIA Corporation, 2025[2025-10-31]. https: / / docs.nvidia.com / nsight-systems / UserGuide / index.html.

[0122] [3] Facebook Research. HolisticTraceAnalysis: Holistic Trace Analysis for Performance Optimization[EB / OL]. Facebook Research, 2020[2025-10-31].https: / / github.com / facebookresearch / HolisticTraceAnalysis.

[0123] [4]PyTorch Team. PyTorch Profiler —PyTorch Tutorials 2.7.0+cu126documentation[EB / OL]. PyTorch Team, 2025[2025-10-31]. https: / / docs.pytorch.org / tutorials / recipes / recipes / profiler_recipe.html.

[0124] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. A method for identifying hotspots in model training that combines cross-thread dependency and multi-critical-path analysis, characterized in that, The method includes: Step 1: Collect the event execution trajectory during the model training process, and parse and standardize the collected raw performance data into a unified format; Step 2: Aggregate the original events through hierarchical modeling and construct edge connections based on dependencies to form a computational dependency graph; Step 3: Identify the sequence of decisive events and the critical paths in the model training process based on the constructed computational dependency graph; Step 4: Through overlap analysis and statistical processing, extract hot events with practical optimization value from the critical path and generate a sorted list.

2. The method as described in claim 1, characterized in that, Step four also includes: providing optimization guidance to developers through dynamic algorithms based on computational dependency graphs and critical path information, and scientifically defining the criteria for determining the optimization endpoint.

3. The method as described in claim 1, characterized in that, In step one, during the information collection process, the capture functions of CPU event tracing, CUDA event tracing, Python call stack, and / or CUDA synchronization events are enabled; And / or, The raw performance data of the collected CPU and GPU event information is parsed into a standardized event sequence containing key fields such as event ID, event name, start time, execution time, and execution thread / CUDA Stream.

4. The method as described in claim 1, characterized in that, In step two, during the event aggregation process, events are divided according to the training phase, including data loading, forward propagation, loss calculation, back propagation, communication, and parameter update; then, CUDA Stream boundary identification is performed, and new nodes are used for GPU events when CUDA Stream switching occurs; events before and after the synchronization operation are marked. And / or, Edge connections in the computational dependency graph are constructed based on four core dependency relationships; And / or, The dependencies include sequential execution dependencies, invocation dependencies, synchronization dependencies, and cross-thread dependencies.

5. The method as described in claim 1, characterized in that, Events from the main thread and the automatic differentiation thread are treated as being executed sequentially within the same logical thread. Dependencies are established based on the actual timestamp order of the events to achieve cross-thread dependency handling, including: Step a. Collect all events in the main thread and the automatic micro-molecule thread, sort them uniformly according to the start timestamp of the events, and form a global event execution sequence; Step b. Establish dependencies between time-adjacent events from the same or different threads; Step c. Add directed edges across thread boundaries in the computation dependency graph from the first executed event to the last executed event.

6. The method as described in claim 1, characterized in that, In step three, an identification strategy based on event timestamps and actual dependencies is used to identify single critical paths and / or multiple critical paths. And / or, In the process of single critical path identification, an endpoint backtracking strategy is adopted. The current node is initialized as the last node to end in the computation dependency graph. Then, in the loop, the path is dynamically selected according to the dependency type between events and the predecessor node with the latest end time is backtracked until the starting point is reached. In the process of identifying multiple critical paths, all predecessor nodes with the same end time are recursively explored, independent path copies are generated, and the operation is repeated to explore and identify all possible paths.

7. The method as described in claim 1, characterized in that, In step four, the statistics on overlapping events between nodes are excluded. The final list of hot events is generated by sorting the time proportion of different events in the critical path. The priority of hot events is evaluated based on their impact on overall performance.

8. The method as described in claim 2, characterized in that, The process is continuously optimized through dynamic algorithms. Based on the changes in performance tracking data after each round of optimization, the critical path is re-identified and the optimization direction is guided. The optimization endpoint is scientifically defined through an event state marking mechanism.

9. A system for implementing the method as described in any one of claims 1-8, characterized in that, The system includes: a performance tracing data acquisition module, a computation dependency graph construction module, a critical path identification module, a hotspot list generation module, and an iterative performance optimization guidance module; The performance tracking data acquisition module collects the event execution trajectory during the training process of the large model and parses the raw performance data into a unified format. The computation dependency graph construction module receives the parsed performance data, adds cross-thread dependency modeling, and constructs an event dependency graph. The critical path identification module, based on the constructed computational dependency graph, enables parallel identification of multiple critical paths; The hotspot list generation module generates a ranking of hotspots based on the critical path analysis results; The iterative performance optimization guidance module dynamically sets optimizations based on the computation dependency graph and critical path information.

10. The method as described in any one of claims 1-8, or the system as described in claim 9, in the characterization and hotspot identification of large model training loads and the optimization of large model training performance.

Images