Large language model training performance analysis method and device, electronic equipment, storage medium and computer program product

Abstract

The invention relates to a big language model training performance analysis method and device, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: aiming at each GPU node which carries out the one-time iteration training of a big language model, carrying out the one-time iteration training of the big language model from the runtime performance data generated by the GPU node, extracting runtime performance data of the plurality of to-be-analyzed algorithm modules; and according to the performance data of the plurality of to-be-analyzed algorithm modules during operation, performing training performance analysis on the iteration training. According to the large language model training performance analysis method provided by the embodiment of the invention, the training performance bottleneck and the optimization direction of the iteration training can be effectively determined.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method and apparatus for analyzing the training performance of large language models, electronic devices, storage media, and computer program products. Background Technology

[0002] In recent years, thanks to the widespread availability of massive datasets and significant improvements in computing resources, Large Language Models (LLMs) have developed rapidly. As the scale of LLMs grows larger, the analysis of their training performance becomes increasingly complex. Therefore, there is an urgent need for a method to quickly and automatically analyze the training performance of LLMs, accelerating the performance optimization and iteration of LLM training. Summary of the Invention

[0003] In view of this, this disclosure presents a method and apparatus for analyzing the training performance of large language models, as well as electronic devices, storage media, and computer program products.

[0004] According to one aspect of this disclosure, a method for analyzing the training performance of a large language model is provided, comprising: for each GPU node performing one iteration of training of a large language model, extracting runtime performance data of multiple algorithm modules to be analyzed from the runtime performance data generated by the GPU node; and performing training performance analysis on the current iteration of training based on the runtime performance data of the multiple algorithm modules to be analyzed.

[0005] In one possible implementation, the step of performing training performance analysis on the current iteration training based on the runtime performance data of the plurality of algorithm modules to be analyzed includes: for each GPU node, determining the time consumed by each algorithm module to be analyzed on the GPU node for each call in the current iteration training based on the runtime performance data of each algorithm module to be analyzed on the GPU node; and performing training performance analysis on the current iteration training based on the time consumed by each algorithm module to be analyzed on the GPU node for each call in the current iteration training.

[0006] In one possible implementation, determining the time consumption of each algorithm module to be analyzed on the GPU node for each GPU node within the current iteration training, based on the runtime performance data of each algorithm module to be analyzed on the GPU node, includes: for the same call of each algorithm module to be analyzed, determining the execution start timestamp of the first GPU core and the execution end timestamp of the last GPU core when the algorithm module to be analyzed completes the call, based on the runtime performance data of the algorithm module to be analyzed completing the call; and determining the time consumption of the algorithm module to be analyzed completing the call based on the execution start timestamp and the execution end timestamp.

[0007] In one possible implementation, the step of performing training performance analysis on each algorithm module to be analyzed on each GPU node based on the time consumption of each call within the current training iteration includes: for each algorithm module to be analyzed, determining the associated parameters for each call completed by the algorithm module to be analyzed based on the runtime performance data of the algorithm module to be analyzed; and for each call of the algorithm module to be analyzed, determining whether there is any time consumption anomaly in the completion of the call by the algorithm module to be analyzed based on the associated parameters for the completion of the call.

[0008] In one possible implementation, when the algorithm module to be analyzed is a computationally intensive function, the correlation parameter is the size of the computation matrix; when the algorithm module to be analyzed is a communication-intensive function, the correlation parameter is the size of the communication data volume.

[0009] In one possible implementation, the training performance analysis of the current iteration training based on the time consumption of each call of each algorithm module to be analyzed on each GPU node within the current iteration training includes: for the same call of each algorithm module to be analyzed, performing statistical analysis on the time consumption of the current call of the algorithm module to be analyzed on different GPU nodes to determine the mean μ and standard deviation σ; and using the 3σ principle, identifying GPU nodes with a time consumption greater than μ+3σ as abnormal GPU nodes.

[0010] In one possible implementation, the step of performing training performance analysis on each algorithm module to be analyzed on each GPU node, based on the time consumed by each call within the current training iteration, includes: statistically analyzing the time consumed by each algorithm module to be analyzed on GPU nodes within the same logical parallel group during the target computation phase, and determining the time consumption percentage of each algorithm module to be analyzed within the logical parallel group; determining the analysis priority of each algorithm module to be analyzed within the logical parallel group based on the time consumption percentage of each algorithm module to be analyzed within the logical parallel group, wherein the time consumption percentage and the analysis priority are directly proportional.

[0011] In one possible implementation, the logical parallel group includes: a pipelined parallel PP stage and different tensor parallel TP groups within the same PP stage.

[0012] In one possible implementation, the target computation phase includes: the current iteration training, the forward computation phase of the current iteration training, and the backward computation phase of the current iteration training.

[0013] According to another aspect of this disclosure, a large language model training performance analysis device is provided, comprising: a data extraction module, configured to extract runtime performance data of multiple algorithm modules to be analyzed from runtime performance data generated by each GPU node performing one iteration of training of a large language model; and a training performance analysis module, configured to perform training performance analysis on the current iteration of training based on the runtime performance data of the multiple algorithm modules to be analyzed.

[0014] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0015] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0016] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0017] In the large language model training performance analysis method of this disclosure embodiment, multiple algorithm modules to be analyzed in one iteration of large language model training are pre-configured according to performance analysis requirements. This allows for the rapid filtering and extraction of runtime performance data of multiple algorithm modules to be analyzed from the massive runtime performance data generated by each GPU node in this iteration of large language model training. Statistical analysis is then performed based on the runtime performance data of multiple algorithm modules to be analyzed to effectively determine the training performance bottleneck and optimization direction of this iteration of training.

[0018] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0019] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0020] Figure 1 A flowchart is shown for a method for analyzing the training performance of a large language model according to an embodiment of the present disclosure.

[0021] Figure 2 A block diagram of a large language model training performance analysis apparatus according to an embodiment of the present disclosure is shown.

[0022] Figure 3 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0023] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0024] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0025] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0026] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0027] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0028] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0029] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0030] In recent years, thanks to the widespread availability of massive datasets and significant improvements in computing resources, large language models have developed rapidly. As the scale of large language models has grown, for example, some have migrated from lower versions with 65 billion parameters to higher versions with 405 billion parameters. Correspondingly, the number of GPUs used for training large language models has also increased dramatically, typically requiring large-scale distributed training on clusters containing thousands to tens of thousands of GPUs. During this process, training algorithms also need to efficiently utilize synchronous / asynchronous operations between CPUs and GPUs, as well as asynchronous mechanisms between multiple streaming command queues within the GPU, to improve training efficiency. Simultaneously, employing complex parallel strategies (e.g., data parallelism (DP), tensor parallelism (TP), pipelined parallelism (PP), etc.) also necessitates frequent synchronization operations between GPU nodes to ensure the coordination and consistency of the global state during distributed training. Furthermore, large language model training often faces limited computing resources and tight time constraints, making the continuous improvement of training efficiency increasingly urgent. However, performance analysis of large language model training is also particularly complex because, in addition to the challenges posed by algorithmic complexity, the deep learning frameworks upon which large language models rely also collect and generate massive amounts of runtime performance data. Therefore, there is an urgent need to integrate and deepen the professional knowledge of large language model training performance analysis into efficient automated analysis tools, so as to quickly find directions for performance improvement.

[0031] In existing technologies, performance analysis methods for large language model training can be divided into two types. One method involves enabling blocking mode, where all operators on the GPU are executed serially. This allows the execution time of all functions to be directly reflected in the function call stack and execution time on the CPU. Since this method only requires analyzing CPU performance data, it simplifies the training performance analysis process. However, because blocking mode transforms some asynchronous operations into synchronous operations, it can lead to a certain degree of distortion in large language model training. The other method considers only the performance analysis at the GPU operator granularity, or performs some stubbing / insertion in the algorithm. However, due to the overly fine / atomic granularity of GPU computation and communication operator information, or insufficient data embedding, or even the lack of a definite analysis direction (making data embedding impractical), a global perspective on training performance analysis and decision-making cannot be formed. The optimization direction becomes localized, resulting in minimal gains.

[0032] Furthermore, existing technologies cannot efficiently process the massive performance data generated during the training of large language models. Improving the training performance of large language models is an iterative process that requires repeated performance analysis and experimental verification. This repetitive process urgently needs efficient and rapid automated training performance analysis methods.

[0033] To address the aforementioned issues, this disclosure provides a method for analyzing the training performance of a large language model. This method can quickly extract and statistically analyze the runtime performance data generated from a single iteration of distributed training of a large language model, effectively identifying the training performance bottlenecks and optimization directions for this distributed training. The method is described in detail below.

[0034] Figure 1 A flowchart illustrating a method for analyzing the training performance of a large language model according to an embodiment of this disclosure is shown. Figure 1 As shown, the method may include:

[0035] In step S11, for each GPU node performing one iteration of training of the large language model, the runtime performance data of multiple algorithm modules to be analyzed is extracted from the runtime performance data generated by the GPU node.

[0036] In step S12, training performance analysis is performed on the current iteration training based on the runtime performance data of multiple algorithm modules to be analyzed.

[0037] In the large language model training performance analysis method of this disclosure embodiment, multiple algorithm modules to be analyzed in one iteration of large language model training are pre-configured according to performance analysis requirements. This allows for the rapid filtering and extraction of runtime performance data of multiple algorithm modules to be analyzed from the massive runtime performance data generated by each GPU node in this iteration of large language model training. Statistical analysis is then performed based on the runtime performance data of multiple algorithm modules to be analyzed to effectively determine the training performance bottleneck and optimization direction of this iteration of training.

[0038] In one example, when a large language model performs distributed training based on a certain training framework (e.g., PyTorch), the training framework can collect runtime performance data generated in each iteration of the large language model training process by setting some parameters.

[0039] For example, with the performance data collection switch of the training framework enabled, distributed training of a large language model can be started. After several training iterations, runtime performance data will be stored in a pre-configured directory path. Taking a training task of a large language model with 1024 GPU nodes (GPU cards) as an example, it is possible to collect the performance trace files (e.g., trace files) generated by the 1024 GPU nodes in each training iteration.

[0040] In one example, the configuration information of the algorithm modules to be analyzed indicates the multiple algorithm modules that need to be analyzed in one iteration of training of the large language model.

[0041] The configuration information for the algorithm modules to be analyzed can be flexibly set according to the actual training performance analysis requirements. For example, for the training performance analysis of the first iteration of training, all functions at the top or several layers above the call stack of the large language model can be configured as the algorithm modules to be analyzed.

[0042] In one example, an algorithm module that is considered to have potential performance issues can be configured as the algorithm module to be analyzed.

[0043] This configurable method of determining the algorithm modules to be analyzed allows users to dynamically define the focus of analysis, rather than using a fixed pattern. That is, ignoring algorithm modules without performance issues and configuring algorithm modules with performance risks in detail, thereby improving the flexibility of performance analysis of large language model training.

[0044] Furthermore, by utilizing this configurable method for determining the algorithm modules to be analyzed, runtime performance data of the algorithm modules to be analyzed can be extracted automatically and in a targeted manner, effectively avoiding the problems of performance data distortion or lack of feasibility caused by using blocking modes and / or cumbersome stubbing or data embedding.

[0045] In one example, the large language model training performance analysis of this disclosure embodiment is a training performance analysis for a single iteration of distributed training of a large language model. After one iteration of training is completed, each GPU node (e.g., a total of 1024 GPU cards) generates an independent Trace file. The Trace file of each GPU node records all CPU-side function call events, GPU-side kernel startup and execution events, and communication events of that GPU node during this iteration of training.

[0046] In one example, for a single iteration of training a large language model, the trace files of each GPU node are processed in parallel to improve the efficiency of training performance analysis. The following describes the processing of the trace files for a single GPU node using this example.

[0047] Based on the trace files of the GPU nodes, a complete function call graph can be extracted from the GPU nodes. Furthermore, based on the configuration information of the algorithm modules to be analyzed, indicating multiple algorithm modules, the complete function call graph can be simplified, and a subgraph of the function call graph for each algorithm module to be analyzed can be flexibly determined.

[0048] Based on the function call relationship subgraph of each algorithm module to be analyzed on the GPU node, the runtime performance data of each algorithm module to be analyzed can be quickly extracted from the massive runtime performance data generated by the GPU node, providing basic data for subsequent training performance analysis.

[0049] In one possible implementation, training performance analysis is performed on the current iteration training based on the runtime performance data of multiple algorithm modules to be analyzed. This includes: for each GPU node, determining the time consumed by each algorithm module to be analyzed on the GPU node for each call during the current iteration training based on the runtime performance data of each algorithm module to be analyzed on the GPU node; and performing training performance analysis on the current iteration training based on the time consumed by each algorithm module to be analyzed on each GPU node for each call during the current iteration training.

[0050] A single training iteration includes a forward computation phase and a backward computation phase. For a given algorithm module to be analyzed, based on its function call relationship subgraph, it's possible to determine whether each call to the algorithm module is in the forward or backward computation phase. Furthermore, the time consumption of each call to the algorithm module on the GPU node can be determined.

[0051] In one possible implementation, for each GPU node, based on the runtime performance data of each algorithm module to be analyzed on the GPU node, the time consumed by each call of each algorithm module to be analyzed on the GPU node within the current iteration training is determined, including: for the same call of each algorithm module to be analyzed, based on the runtime performance data of the algorithm module to be analyzed completing this call, determining the execution start timestamp of the first GPU core and the execution end timestamp of the last GPU core when the algorithm module to be analyzed completes this call; and determining the time consumed by the algorithm module to be analyzed to complete this call based on the execution start timestamp and the execution end timestamp.

[0052] Mapping the execution time of GPU kernel functions to the CPU-side function call graph, the upper-level CPU-side functions will launch a large number of GPU kernel functions. The time consumed by each CPU-side function (the algorithm module to be analyzed) in a single call should cover the time consumed from triggering the first GPU kernel to the end of the last GPU kernel execution.

[0053] For a single call to an algorithm module to be analyzed, in the function call relationship subgraph of the algorithm module to be analyzed, determine the execution start time stamp t1 of the first GPU core triggered by this call and the execution end time stamp t of the last GPU core, and then determine the duration of this call as: Duration = t2 - t1.

[0054] In one possible approach, training performance analysis is performed on each iteration of training based on the time consumed by each call to the algorithm module to be analyzed on each GPU node within the current training iteration. This includes: for each algorithm module to be analyzed, determining the associated parameters for each call to be completed by the algorithm module to be analyzed based on the runtime performance data of the algorithm module to be analyzed; and for each call to the algorithm module to be analyzed, determining whether there are any time consumption anomalies in the completion of the call by the algorithm module to be analyzed based on the associated parameters for the completion of the call.

[0055] For each GPU node, after determining the time taken for each algorithm module to be analyzed to complete each call on that GPU node, the associated parameters for each call are extracted from the runtime performance data of that algorithm module. For a single call of an algorithm module to be analyzed, the time determination and associated parameter extraction are performed simultaneously to ensure that the time taken for each call is accurately bound to its associated parameters (computation / communication context).

[0056] In one example, the configuration information of the algorithm module to be analyzed can indicate the type of each algorithm module to be analyzed, to indicate whether the algorithm module to be analyzed is a computationally intensive function or a communication-intensive function.

[0057] In one possible implementation, if the algorithm module to be analyzed is a computationally intensive function, the associated parameter is the size of the computation matrix; if the algorithm module to be analyzed is a communication-intensive function, the associated parameter is the size of the communication data volume.

[0058] When the algorithm module to be analyzed is a computationally intensive function, the associated parameters of the algorithm module to be analyzed for each call are the size of the computation matrix involved in the calculation when the algorithm module to be analyzed completes this call.

[0059] For example, for a single call to a computationally intensive function, the dimensions (shapes) of multiple matrices passed in during each call are collected, and the average matrix size is calculated as a related parameter: the matrix size is calculated.

[0060] When the algorithm module to be analyzed is a computationally intensive function, the theoretical number of floating-point operations (FLOPs) to be completed by the algorithm module to be analyzed in this call is determined based on the size of the computation matrix of the algorithm module to be analyzed in this call. Then, combined with the theoretical peak computing power of the GPU node, the theoretical time range of the algorithm module to be analyzed in this call is determined, which serves as the theoretical basis for judging whether there is any abnormality in the actual time of the algorithm module to be analyzed in this call.

[0061] When the algorithm module to be analyzed is a communication-intensive function, the associated parameters of each call of the algorithm module to be analyzed represent the amount of communication data involved in the communication when the algorithm module to be analyzed completes this call.

[0062] For example, for a single call to a communication-intensive function, the dimensions (shapes) of multiple matrices passed in during each call will be collected, and the average size of the matrices will be calculated as a correlation parameter: the amount of communication data.

[0063] When the algorithm module to be analyzed is a communication-intensive function, the theoretical time range of the algorithm module to be analyzed for this call is determined based on the amount of communication data completed by the algorithm module to be analyzed for this call, combined with the network topology and the theoretical bandwidth of the link. This serves as the theoretical basis for judging whether there is any abnormality in the actual time taken by the algorithm module to be analyzed for this call.

[0064] For a call to an algorithm module to be analyzed on a GPU node, if the actual time taken for the algorithm module to complete the call exceeds the theoretical time range, it is determined that there is an anomaly in the time taken for the algorithm module to complete the call. This can be identified as a bottleneck in the computation operator or communication operator of the current iteration training, or insufficient overlap between communication and computation, which can be used as a direction for algorithm optimization and analysis.

[0065] For a GPU node, if an algorithm module to be analyzed has an abnormal time consumption only in a single call, while the time consumption of other calls is normal, it can be determined that there is a momentary interference (e.g., system scheduling, contention for shared resources), or that there is a special data path problem in this call.

[0066] For a GPU node, if the associated parameters of an algorithm module to be analyzed are normal in multiple consecutive calls, but the time consumption is abnormal, it can be determined that the GPU node has hardware performance degradation or problems such as software scheduling / GPU kernel inefficiency.

[0067] Since large language models are composed of multiple identical Transformer Layers stacked together, the same algorithm module to be analyzed (e.g., SelfAttention) will be called multiple times in one iteration of training (corresponding to different model layers). Therefore, the time consumption of each call to the same algorithm module to be analyzed, the same computation stage, and different model layers is determined separately to obtain the time consumption array [d1,d2,d3...] of the algorithm module to be analyzed in that computation stage.

[0068] In one possible implementation, training performance analysis is performed on each iteration of training based on the time taken for each algorithm module to be analyzed on each GPU node during each call within the current training iteration. This includes: statistically analyzing the time taken for the same call of each algorithm module to be analyzed on different GPU nodes to determine the mean μ and standard deviation σ; and using the 3σ principle, identifying GPU nodes with a time taken greater than μ+3σ as abnormal GPU nodes.

[0069] For the same call of each algorithm module to be analyzed, after determining the time taken by the algorithm module to be analyzed on different GPU nodes, the health of GPU nodes in the cluster can be analyzed by comparing the time taken on different GPU nodes, and abnormal GPU nodes with hardware failure, poor heat dissipation or severe performance degradation can be quickly located.

[0070] For example, for the same call to the algorithm module to be analyzed: "the SelfAttention module's call in the forward computation phase, Layer 5", the mean μ and standard deviation σ are calculated based on the time taken by the algorithm module to be analyzed in this call on each GPU node. Then, statistical rules (e.g., the 3σ principle) are applied to identify GPU nodes with a time taken greater than μ + 3σ as abnormal GPU nodes.

[0071] In one example, abnormal GPU nodes in the cluster are marked to indicate subsequent fault repair.

[0072] In one possible implementation, training performance analysis is performed on the current iteration of training based on the time consumed by each algorithm module to be analyzed on each GPU node during each call within the current iteration. This includes: statistically analyzing the time consumed by each algorithm module to be analyzed on GPU nodes within the same logical parallel group during a single computation phase to determine the time consumption percentage of each algorithm module to be analyzed within the logical parallel group; and determining the analysis priority of each algorithm module to be analyzed within the logical parallel group based on the time consumption percentage of each algorithm module to be analyzed within the logical parallel group, wherein the time consumption percentage and the analysis priority are directly proportional.

[0073] A logical parallel group refers to a group of GPU nodes in distributed training that perform the exact same logical task due to a parallel strategy. GPU nodes within the same logical parallel group run the same model code and the same layers, with completely identical task logic.

[0074] In one possible implementation, the logical parallel group includes: pipelined parallel PP stages and different tensor parallel TP groups within the same PP group.

[0075] Depending on the model size, the parallel strategy for distributed training can be to divide the training into multiple parallel PP stages, or to further divide the training into multiple parallel TP groups within the same PP stage. The specific parallel method can be flexibly set according to the size of the large language model, and this disclosure does not impose any specific limitations on it.

[0076] For example, in training with PP=4 and TP=8, a logical parallel group can be defined as (PP_Stage=i, TP_Group=j), where i∈[0,3] and j∈[0,7]. In one training iteration, there are 32 logical parallel groups. The number of GPU nodes in each logical parallel group is determined by the data parallelism (DP) dimension. All GPU nodes within each logical parallel group collaborate to complete specific slice computations of a particular part of the large language model (e.g., specific layers).

[0077] In one example, by comparing the results horizontally, abnormally time-consuming data in this iteration of training can be marked or removed to ensure that the calculation of the subsequent time consumption ratio is not distorted by individual abnormally time-consuming data.

[0078] For the same logical parallel group (e.g., PP_Stage2, TP_Group1) and the selected target computation stage, perform the following time percentage calculation.

[0079] In one possible implementation, the target computation phase includes: the current iteration training, the forward computation phase of the current iteration training, and the backward computation phase of the current iteration training.

[0080] Depending on the granularity of the training performance analysis, the time consumption percentage can be analyzed throughout the entire training iteration, or it can be further refined to analyze the time consumption percentage in the forward computation phase and the backward computation phase of the training iteration.

[0081] For each algorithm module to be analyzed on each GPU node within the same logical parallel group that is in the target computation stage, the aforementioned time consumption calculation method can be used to determine the time consumption array [d1,d2,d3...] (corresponding to calls at different layers) of the algorithm module to be analyzed.

[0082] For a given algorithm module to be analyzed, based on the time consumption array of the algorithm module on each GPU node, the maximum, minimum, and average time consumption of the algorithm module to be analyzed within the logical parallel group during the target computation stage can be determined.

[0083] Furthermore, based on the average time consumption and total number of calls of the algorithm module to be analyzed within the logical parallel group during the target computation stage, the total contribution time of the algorithm module to be analyzed within the logical parallel group during the target computation stage can be determined.

[0084] The total contribution time of each algorithm module to be analyzed within the logical parallel group in the target computation stage is summed to obtain the baseline total time of the logical parallel group in the target computation stage.

[0085] For each algorithm module to be analyzed within the logical parallel group in the target computing stage, the time consumption percentage of the algorithm module to be analyzed within the logical parallel group in the target computing stage is calculated based on the total contribution time of the algorithm module to be analyzed and the baseline total time of the logical parallel group in the target computing stage.

[0086] For example, within the logical parallel groups (PP_Stage2, TP_Group1) of the target computation stage (forward computation stage), after the above calculations, it is determined that the time consumption of the algorithm module AllReduce to be analyzed accounts for 45%, the time consumption of the algorithm module SelfAttention to be analyzed accounts for 30%, the time consumption of the algorithm module MLP to be analyzed accounts for 20%, and the time consumption of other algorithm modules to be analyzed accounts for 5%.

[0087] After determining the time consumption percentage of each algorithm module to be analyzed within the same logical parallel group in the target computation stage, the analysis priority of each algorithm module to be analyzed within that logical parallel group in the target computation stage can be determined.

[0088] The higher the time consumption percentage of the algorithm module to be analyzed, the worse its performance, and therefore, the higher its analysis priority. Within the same logical parallel group of the target computation stage, the algorithm module to be analyzed with the highest time consumption percentage is the primary performance bottleneck within that logical parallel group of the target computation stage.

[0089] Based on the aforementioned method, it is possible to statistically determine the maximum, minimum, average, percentage, and analysis priority of each algorithm module to be analyzed within the same logical parallel group during the target computation stage.

[0090] Furthermore, based on the statistical data, a more detailed analysis of the training performance can be conducted. For example, based on the maximum, minimum, and average execution time of the algorithm module to be analyzed, the volatility of the algorithm module can be determined as (maximum execution time - minimum execution time) / average execution time. The volatility can measure the degree of dispersion in the execution time of the algorithm module to be analyzed.

[0091] In addition to the volatility analysis mentioned above, other analyses can be performed according to the actual training performance analysis needs, and this disclosure does not impose specific limitations on them.

[0092] Based on the large language model training performance analysis method disclosed herein, the training performance bottlenecks and optimization directions of the current iteration are identified. After optimizing the large language model training, iterative training continues. During the training performance analysis of the new round of iterations after optimization, the new runtime performance data can quickly refresh the performance data and proportions of algorithm call relationships, directly reflecting whether the optimization scheme meets expectations. For example, whether the actual time consumption of the corresponding algorithm modules under analysis has yielded corresponding optimization benefits. Simultaneously, the training performance analysis of the new round of iterations can also identify the corresponding training performance bottlenecks and optimization directions again. Continuously iterating this process can accelerate the performance ramp-up of the large language model training.

[0093] This disclosure presents a method for analyzing the training performance of large language models, enhancing the value of automation in the training performance analysis process. It improves efficiency while avoiding tedious and accidental low-level operations introduced by repetitive manual processes. Furthermore, the automation of the entire training performance analysis process optimizes and avoids repetitive manual handling.

[0094] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0095] In addition, this disclosure also provides a large language model training performance analysis device, electronic device, computer-readable storage medium, and program, all of which can be used to implement any of the large language model training performance analysis methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section and will not be repeated here.

[0096] Figure 2 A block diagram of a large language model training performance analysis apparatus according to an embodiment of the present disclosure is shown. Figure 2 As shown, the device 20 includes:

[0097] The data extraction module 21 is used to extract runtime performance data of multiple algorithm modules to be analyzed from the runtime performance data generated by the GPU node for each GPU node performing one iteration of training of the large language model.

[0098] The training performance analysis module 22 is used to perform training performance analysis on the current iteration training based on the runtime performance data of multiple algorithm modules to be analyzed.

[0099] In one possible implementation, the training performance analysis module 22 is specifically used for:

[0100] For each GPU node, based on the runtime performance data of each algorithm module to be analyzed on the GPU node, determine the time consumed by each call of each algorithm module to be analyzed on the GPU node in this iteration of training;

[0101] Based on the time consumed by each call to each algorithm module to be analyzed on each GPU node within this iteration of training, the training performance of this iteration is analyzed.

[0102] In one possible implementation, the training performance analysis module 22 is specifically used for:

[0103] For the same call of each algorithm module to be analyzed, based on the runtime performance data of the algorithm module to be analyzed when it completes the call, determine the execution start timestamp of the first GPU core and the execution end timestamp of the last GPU core when the algorithm module to be analyzed completes the call;

[0104] Based on the execution start timestamp and execution end timestamp, determine the time taken for the algorithm module to be analyzed to complete this call.

[0105] In one possible implementation, the training performance analysis module 22 is specifically used for:

[0106] For each algorithm module to be analyzed, the associated parameters for each call to be completed by the algorithm module to be analyzed are determined based on the runtime performance data of the algorithm module to be analyzed.

[0107] For each call to the algorithm module to be analyzed, based on the associated parameters of the algorithm module to be analyzed to complete this call, it is determined whether there is any time consumption anomaly in the algorithm module to be analyzed to complete this call.

[0108] In one possible implementation, when the algorithm module to be analyzed is a computationally intensive function, the associated parameter is the size of the computation matrix;

[0109] When the algorithm module to be analyzed is a communication-intensive function, the associated parameter is the amount of communication data.

[0110] In one possible implementation, the training performance analysis module 22 is specifically used for:

[0111] For the same call of each algorithm module to be analyzed, statistical analysis is performed on the time consumption of this call of the algorithm module to be analyzed on different GPU nodes to determine the mean μ and standard deviation σ;

[0112] Using the 3σ principle, GPU nodes that take longer than μ+3σ are identified as abnormal GPU nodes.

[0113] In one possible implementation, the training performance analysis module 22 is specifically used for:

[0114] For GPU nodes within the same logical parallel group, the time consumed by each algorithm module to be analyzed during the target computation stage is statistically analyzed to determine the time consumption percentage of each algorithm module to be analyzed within the logical parallel group.

[0115] The analysis priority of each algorithm module to be analyzed in the logical parallel group is determined based on the time consumption percentage of each algorithm module to be analyzed in the logical parallel group. The time consumption percentage is directly proportional to the analysis priority.

[0116] In one possible implementation, the logical parallel group includes: PP stage and different TP groups within the same PP stage.

[0117] In one possible implementation, the target computation phase includes: the current iteration training, the forward computation phase of the current iteration training, and the backward computation phase of the current iteration training.

[0118] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0119] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0120] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0121] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0122] Figure 3A block diagram of an electronic device according to this embodiment is shown. (Refer to...) Figure 3 The electronic device 1900 can be provided as a server or a terminal device. (See reference...) Figure 3 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0123] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0124] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0125] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0126] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0127] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0128] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0129] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0130] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0131] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0132] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for analyzing the training performance of a large language model, characterized in that, include: For each GPU node performing one iteration of training of a large language model, the runtime performance data of multiple algorithm modules to be analyzed is extracted from the runtime performance data generated by the GPU node. Based on the runtime performance data of the multiple algorithm modules to be analyzed, the training performance of this iteration is analyzed.

2. The method according to claim 1, characterized in that the iteration, specifically the step of performing training performance analysis on the current iteration training based on the runtime performance data of the plurality of algorithm modules to be analyzed, includes: For each GPU node, based on the runtime performance data of each algorithm module to be analyzed on the GPU node, determine the time consumed by each call of each algorithm module to be analyzed on the GPU node in this iteration of training; Based on the time consumed by each call to each algorithm module to be analyzed on each GPU node within this iteration of training, the training performance of this iteration is analyzed.

3. The method according to claim 2, characterized in that, For each GPU node, based on the runtime performance data of each algorithm module to be analyzed on the GPU node, the time consumed by each call to the algorithm module to be analyzed on the GPU node within this iteration of training is determined, including: For the same call of each algorithm module to be analyzed, based on the runtime performance data of the algorithm module to be analyzed when it completes the call, determine the execution start time of the first GPU core and the execution end time of the last GPU core when the algorithm module to be analyzed completes the call; Based on the execution start timestamp and the execution end timestamp, the time taken for the algorithm module to be analyzed to complete this call is determined.

4. The method according to claim 2, characterized in that, The training performance analysis is performed on each iteration of training based on the time consumed by each algorithm module to be analyzed on each GPU node during the current training iteration, including: For each algorithm module to be analyzed, the associated parameters for each call to be completed by the algorithm module to be analyzed are determined based on the runtime performance data of the algorithm module to be analyzed. For each call to the algorithm module to be analyzed, based on the associated parameters of the algorithm module to be analyzed in completing this call, it is determined whether there is any time consumption abnormality in the algorithm module to be analyzed in completing this call.

5. The method according to claim 4, characterized in that, When the algorithm module to be analyzed is a computationally intensive function, the correlation parameter is the size of the computation matrix; When the algorithm module to be analyzed is a communication-intensive function, the correlation parameter is the amount of communication data.

6. The method according to claim 2, characterized in that, The training performance analysis is performed on each iteration of training based on the time consumed by each algorithm module to be analyzed on each GPU node during the current training iteration, including: For the same call of each algorithm module to be analyzed, statistical analysis is performed on the time consumption of this call of the algorithm module to be analyzed on different GPU nodes to determine the mean μ and standard deviation σ; Using the 3σ principle, GPU nodes that take longer than μ+3σ are identified as abnormal GPU nodes.

7. The method according to claim 2, characterized in that, The training performance analysis is performed on each iteration of training based on the time consumed by each algorithm module to be analyzed on each GPU node during the current training iteration, including: For GPU nodes within the same logical parallel group, the time consumed by each algorithm module to be analyzed during the target computation stage is statistically analyzed to determine the time consumption percentage of each algorithm module to be analyzed within the logical parallel group. The analysis priority of each algorithm module to be analyzed in the logical parallel group is determined based on the time consumption ratio of each algorithm module to be analyzed in the logical parallel group, wherein the time consumption ratio is directly proportional to the analysis priority.

8. The method according to claim 7, characterized in that, The logical parallel group includes: pipelined parallel PP stage and different tensor parallel TP group within the same PP stage.

9. The method according to claim 7, characterized in that, The target computation phase includes: the current iteration training, the forward computation phase of the current iteration training, and the backward computation phase of the current iteration training.

10. A large language model training performance analysis device, characterized in that, include: The data extraction module is used to extract runtime performance data of multiple algorithm modules to be analyzed from the runtime performance data generated by each GPU node performing one iteration of training of a large language model. The training performance analysis module is used to perform training performance analysis on the current iteration training based on the runtime performance data of the multiple algorithm modules to be analyzed.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

12. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images