Deep learning model training performance analysis method and device, equipment and medium

Abstract

The embodiment of the invention provides a training performance analysis method and device of a deep learning model, electronic equipment and a computer readable storage medium. The method specifically comprises the steps of obtaining first operation data of an operator in a deep learning model in different dimensions under the condition that the deep learning model is trained; the deep learning model comprises at least two operators; associating the first operation data of the operator in different dimensions through a preset association algorithm to obtain joint operation data of the operator; and according to the joint operation data, performing training performance analysis on the deep learning model to obtain a training performance analysis result of the deep learning model. According to the method, the incidence relation between operation data of operators in different dimensions can be established, correlation between the operation data of the operators and a pipeline structure is achieved, a data basis is provided for performance analysis of a model and positioning of an abnormal condition, and the efficiency of model training performance analysis is improved; and the accuracy of a model training performance analysis result is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and computer-readable storage medium for analyzing the training performance of a deep learning model. Background Technology

[0002] With the rapid increase in the parameter size of large models, the computational resources, storage capacity, and training time required for training also increase accordingly. Hybrid parallel training has become a key technique for training large models. In practical training scenarios, it is necessary to analyze the performance of large model training.

[0003] In related technologies, performance analysis data is usually obtained through manual collection for offline performance analysis.

[0004] However, the offline analysis method in related technologies is time-consuming, resulting in low efficiency in model performance analysis, and the quantitative accuracy of large model training performance in related technologies is insufficient. Summary of the Invention This application provides a method, apparatus, electronic device, and computer-readable storage medium for analyzing the training performance of deep learning models, which can improve the efficiency of analyzing the training performance of large models.

[0005] Firstly, this application provides a method for analyzing the training performance of a deep learning model, the method comprising: When training a deep learning model, obtain the first running data of the operators in the deep learning model in different dimensions; the deep learning model includes at least two operators. By using a pre-defined association algorithm, the first running data of the operator in different dimensions are associated to obtain the joint running data of the operator; Based on the joint operation data, the training performance of the deep learning model is analyzed, and the training performance analysis results of the deep learning model are obtained.

[0006] Secondly, this application provides a training performance analysis device for a deep learning model, the device comprising: an acquisition module, an association module, and an analysis module.

[0007] The acquisition module is used to acquire the first running data of the operators in the deep learning model in different dimensions during the training of the deep learning model; the deep learning model includes at least two operators; The association module is used to associate the first running data of the operator in different dimensions through a preset association algorithm to obtain the joint running data of the operator; The analysis module is used to perform training performance analysis on the deep learning model based on the joint operation data, and obtain the training performance analysis results of the deep learning model.

[0008] Thirdly, this application provides an electronic device including a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors as described above for training performance analysis of a deep learning model.

[0009] Fourthly, this application provides a readable storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the training performance analysis method for the deep learning model as described above.

[0010] Fifthly, this application provides a computer program product comprising a computer program that is loaded and executed by a processor to implement the training performance analysis method for the deep learning model as described above.

[0011] The embodiments of this application have the following advantages: This application's embodiments first acquire the first running data of operators in different dimensions of the deep learning model during training. This allows for the collection of operator-level performance data, refining the granularity of performance analysis data and improving the accuracy of subsequent performance analysis results. Simultaneously, real-time data acquisition during model training ensures that the collected data reflects the model's true training state, enabling timely identification and response to dynamically emerging performance issues and avoiding analytical biases caused by data lag. For example, an offline acquisition and analysis approach involves exporting logs after model training. If the GPU experiences a sudden 10-millisecond load peak, the log might not capture this data due to the sampling interval, leading to biases in model training performance analysis. Secondly, a pre-defined association algorithm is used to correlate the first running data of operators in different dimensions, obtaining joint running data for the operators. Finally, based on this joint running data, training performance analysis of the deep learning model is performed, yielding the training performance analysis results. Furthermore, a pipeline parallel strategy can be used to correlate the first running data after unifying the timeline, obtaining joint data and establishing a spatiotemporal correlation between the operator's running data in different dimensions. This establishes a connection between operator running data and the pipeline structure, providing a data foundation for model performance analysis and anomaly localization. When analyzing the joint running data, all related information can be clearly obtained, reducing the time spent on model training performance analysis and thus improving the efficiency of model training performance analysis. At the same time, the joint running data includes the first running data of the operator in different dimensions and the related information of each first running data. Performance analysis based on the joint running data can improve the accuracy of the model training performance analysis results. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of a pipeline parallelism provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the steps of a method for analyzing the training performance of a deep learning model, as provided in an embodiment of this application. Figure 3 This is a schematic diagram of the structure of a deep learning model training performance analysis system provided in an embodiment of this application; Figure 4 This is a flowchart illustrating the steps of another deep learning model training performance analysis method provided in this application embodiment; Figure 5 This is a comparative diagram of simulated operation data and combined operation data provided in an embodiment of this application; Figure 6 This is a schematic diagram illustrating the specific implementation process of a deep learning model training performance analysis method provided in an embodiment of this application; Figure 7 This is a structural block diagram of a deep learning model training performance analysis device provided in an embodiment of this application; Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and are not limited in number; for example, a first object can be one or more. Furthermore, the term "and / or" in the specification and claims is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0016] With the rapid increase in the parameter size of large models, the computational resources, storage capacity, and training time required for training also increase accordingly. Hybrid parallel training has become a key technique for training large models. In practical training scenarios, it is necessary to analyze the performance of large model training.

[0017] In related technologies, performance analysis data is usually obtained through manual collection for offline performance analysis.

[0018] However, offline analysis methods in related technologies are time-consuming, resulting in low efficiency in model performance analysis.

[0019] For example, with the rapid growth of large model parameter scale, the computational resources, storage capacity, and training time required for training increase accordingly. Hybrid parallel training has become a key technology to address these challenges. Pipeline parallelism divides the model into multiple stages, each distributed across different computing devices. Training data is further divided into multiple micro-batches, with computing devices continuously and actively executing in a pipelined manner. This not only effectively alleviates the memory pressure caused by large models but also significantly improves resource utilization on individual devices. In real-world training scenarios, cavitation may occur during pipeline execution, resulting in significant idle waiting time between devices, which can slow down the entire training process or cause performance anomalies. Uneven load distribution between devices can lead to some devices being idle, reducing overall computational efficiency. Therefore, it is necessary to analyze the performance of large model training using pipeline parallelism.

[0020] For example, Figure 1 This is a schematic diagram of a parallel pipeline provided in an embodiment of this application; as shown... Figure 1As shown, taking the example of each computing device being responsible for one stage, the model is divided into two stages by layer, namely Stage1 and Stage2; each device is responsible for one stage, and the data is processed in two micro-batches, namely micro-batch 1 (MB1) and micro-batch 2 (MB2).

[0021] During the forward computation process, MB1 is input to device 1. Device 1 performs the first stage of forward computation on MB1. After completing the first stage (Stage 1) of forward computation on MB1, device 1 sends the result of the forward computation of MB1 in Stage 1 to device 2. At the same time, device 1 does not wait and continues to perform the forward computation of MB2 in Stage 1. After receiving the result of the forward computation of MB1 in Stage 1 sent by device 1, device 2 performs the forward computation of MB1 in the second stage (Stage 2) based on the result. The forward computation of MB1 in Stage 2 by device 2 and the forward computation of MB2 in Stage 1 by device 1 are performed in parallel. After completing the forward computation of MB1 in Stage 2, device 2 sends the result of the forward computation of MB1 in Stage 2 to device 1. The forward computation process of MB2 by device 1 and device 2 is the same as the forward computation process of MB1 by device 1 and device 2.

[0022] During the reverse computation process, backpropagation relies on the "gradient of the next stage," therefore it starts from Stage 2 and processes the last micro-batch first. Device 2 performs the reverse computation of MB2 in Stage 2 based on the forward computation result of MB2 in Stage 2, and sends the result to Device 1. Simultaneously, Device 2 continues its reverse computation of MB1 in Stage 2 without waiting. After receiving the reverse computation result of MB2 in Stage 2 from Device 2, Device 1 performs the reverse computation of MB2 in Stage 1 based on that result. The reverse computation of MB2 in Stage 1 by Device 1 and the reverse computation of MB1 in Stage 1 by Device 2 are performed concurrently. This micro-batch pipeline approach ensures that the computing devices at different stages operate as continuously as possible.

[0023] In the parallel implementation of the pipeline described above, device 2 will only perform the forward calculation of MB1 in Stage 2 after receiving the forward calculation result of MB1 in Stage 1 sent by device 1. Therefore, before receiving the forward calculation result of MB1 in Stage 1 sent by device 1, device 2 is in an idle state, meaning a cavitation occurs during pipeline execution. After device 2 completes the reverse calculation of MB1 in Stage 2, device 2 is in an idle state, meaning a cavitation occurs during pipeline execution. Device 1 will only perform the reverse calculation of MB2 in Stage 1 after receiving the reverse calculation result of MB2 in Stage 2 sent by device 2; therefore, before receiving the reverse calculation result of MB2 in Stage 2 sent by device 2, device 1 is in an idle state, meaning a cavitation occurs during pipeline execution.

[0024] In related technologies, performance analysis of large model training typically requires manual collection and acquisition of performance data for offline analysis to pinpoint performance issues. This approach necessitates initiating performance data collection before analysis, and incurs significant overhead during data collection. It struggles to support real-time dynamic analysis and cannot promptly identify and respond to dynamically arising performance issues during training. For example, dynamic issues such as uneven load distribution in the pipeline or sudden changes in sequence length may not be addressed in time due to the lag in offline analysis, hindering guidance for dynamic optimization.

[0025] Furthermore, many related technologies focus on the single-machine or single-device layer, lacking the ability to analyze the global parallel performance between nodes and processes (rank), making it difficult to quickly locate performance bottlenecks in distributed training execution. For example, in pipelined parallelism, bubbles caused by inter-device dependencies lead to idle computing resources. The analysis methods in related technologies are difficult to quantify the bubble ratio, usually only providing operator-level time statistics, which cannot correlate operator execution with pipeline structure, resulting in increased time consumption for problem localization.

[0026] Against this background, embodiments of this application provide a method for analyzing the training performance of a deep learning model, which can reduce the time spent on model performance analysis and improve the efficiency of model training performance analysis.

[0027] The training performance analysis method of the deep learning model according to the embodiments of this application will be further described below with reference to the relevant accompanying drawings and examples: Figure 2 This is a flowchart illustrating the steps of a method for analyzing the training performance of a deep learning model, as provided in an embodiment of this application. Figure 2 As shown, the method may include steps 201-203.

[0028] Step 201: While the deep learning model is being trained, obtain the first running data of the operators in the deep learning model in different dimensions.

[0029] In this embodiment, the deep learning model may include at least two operators. In the scenario of training a deep learning model, an operator (Op) is the basic computational unit for model training and inference, responsible for performing a single, specific computation or data processing operation. It is an atomic component constituting a model layer, stage, or even the entire distributed training task. Based on the actual training scenario of the deep learning model, first running data of at least one operator in the deep learning model in different dimensions can be obtained.

[0030] For example, the first running data of each operator in the deep learning model in different dimensions can be obtained according to the actual training scenario of the deep learning model, or the first running data of some operators in the deep learning model in different dimensions can be obtained according to the actual training scenario of the deep learning model.

[0031] The dimensions can include time and space. The first operational data of an operator in the time dimension can include the operator's timestamp, the operator's execution time, and the timestamps of the micro-batch data processed by the operator; that is, the operator's timestamp, the operator's execution time, and the timestamps of the micro-batch data processed by the operator all belong to the operator's first operational data in the time dimension. The first operational data of an operator in the space dimension can include information about the actual hardware device on which the operator runs, the pipeline stage identifier where the operator is located, the communication link of the pipeline stage where the operator is located, and the identifiers of the micro-batch data processed by the operator; that is, the information about the actual hardware device on which the operator runs, the pipeline stage identifier where the operator is located, the communication link of the pipeline stage where the operator is located, and the identifiers of the micro-batch data processed by the operator all belong to the operator's first operational data in the space dimension.

[0032] Figure 3 This is a schematic diagram of the structure of a deep learning model training performance analysis system provided in an embodiment of this application. For example... Figure 3As shown, in a distributed training scenario, this training performance analysis system can include multiple worker nodes (e.g., Worker1~N). Each node can deploy a data acquisition agent, which resides in the training process and performs real-time performance monitoring of all computing devices during training. The data acquisition agent can use a non-blocking collector (e.g., a PyTorch-based hook function) to collect and record operator-level runtime data in real time during the pipelined parallel training of the model. This operator-level runtime data can specifically include operator execution time, memory usage, central processing unit (CPU) utilization, graphics processing unit (GPU) utilization, network input / output (I / O), etc. In other words, the data acquisition agent can capture instantaneous operator behaviors without omission using a non-blocking collector. Then, it extracts the first runtime data of each operator in different dimensions from the aforementioned real-time recorded data. The training performance analysis system may also include a central master module. Data extracted by the data acquisition agent can be sent from each working node to the central master in batches, and the central master will further process the data.

[0033] Step 202: Using a preset association algorithm, associate the first running data of the operator in different dimensions to obtain the joint running data of the operator.

[0034] In this embodiment, the preset association algorithm may include a spatiotemporal association modeling algorithm. After collecting first running data of at least one operator from multiple working nodes in different dimensions, the first running data of the operator in the time dimension can be aligned with a unified time axis (e.g., global clock synchronization / logical clock). After aligning the first running data of the operator in the time dimension with a unified time axis, when training the model in a pipelined parallel manner, the pipelined parallel strategy for model training can be obtained. The spatiotemporal association modeling algorithm then associates the first running data of each operator from multiple working nodes in different dimensions according to the pipelined parallel strategy to obtain the joint running data of each operator. The joint running data can be understood as: in a distributed training scenario of a deep learning model, the structured set of running state data from a global perspective obtained after spatiotemporally associating the running data generated by the operators in the deep learning model.

[0035] For example, the timestamps (or event sequences) of the first run data of an operator can be mapped to the same baseline time dimension. Taking global clock synchronization as an example, in a 3-node pipeline parallel training, if the stage containing operator 1 is deployed on node 1, the stage containing operator 2 is deployed on node 2, and the stage containing operator 3 is deployed on node 3, then a node can be understood as a hardware device. Node 1 locally acquires the start timestamp of operator 1 at 200ms, and its clock is calculated to be 50ms ahead of the global reference (i.e., the offset is +50ms). Therefore, the global timestamp is 100-50=50ms. Node 2 locally acquires the start timestamp of operator 2 at 150ms, and its clock offset is +50ms. The global timestamp is 150-50=100ms. Node 3 locally acquires the start timestamp of operator 3 at 220ms, and its clock offset is -30ms. The global timestamp is 220+30=250ms. After correction, the global timestamps of the three operators form a unified timing sequence, which can be directly used for subsequent association.

[0036] After aligning the first running data of the operator in the time dimension with a unified time axis, taking operator 1 as an example, the start timestamp of operator 1 is 100ms and the end timestamp is 200ms. The start timestamp of micro-batch data 1 is 100ms and the end timestamp is 200ms. Then, operator 1 and micro-batch data 1 are associated.

[0037] The pipeline parallel strategy may include: the model includes stage 1 and stage 2, stage 1 is processed by hardware device 1, stage 2 is processed by hardware device 2, stage 1 includes operator 1 and operator 4, stage 2 includes operator 2 and operator 5, stage 1 and stage 2 communicate through communication link 1. Based on the information contained in the above pipeline parallel strategy, operator 1, stage 1, hardware device 1 and communication link 1 can be associated. Since operator 1 and micro-batch data 1 have been associated, operator 1, micro-batch data 1, stage 1, hardware device 1 and communication link 1 can be associated through spatiotemporal association modeling algorithm to obtain the joint operation data of operator 1.

[0038] Step 203: Based on the joint operation data, perform training performance analysis on the deep learning model to obtain the training performance analysis results of the deep learning model.

[0039] In this embodiment, after obtaining the joint running data, the training performance of the deep learning model can be analyzed based on the joint running data to obtain the training performance analysis results of the deep learning model. For example, the average computation time of all operators in a certain stage can be calculated using the joint running data, and the training performance of the deep learning model can be analyzed based on this average computation time combined with historical data; as another example, the utilization rate of a certain GPU can be calculated using the joint running data, and the training performance of the deep learning model can be analyzed based on this utilization rate combined with the operators associated with the GPU and other data.

[0040] This application's embodiments first acquire the first running data of operators in different dimensions of the deep learning model during training. This allows for the collection of operator-level performance data, refining the granularity of performance analysis data and improving the accuracy of subsequent performance analysis results. Simultaneously, real-time data acquisition during model training ensures that the collected data reflects the model's true training state, enabling timely identification and response to dynamically emerging performance issues and avoiding analytical biases caused by data lag. For example, an offline acquisition and analysis approach involves exporting logs after model training. If the GPU experiences a sudden 10-millisecond load peak, the log might not capture this data due to the sampling interval, leading to biases in model training performance analysis. Secondly, a pre-defined association algorithm is used to correlate the first running data of operators in different dimensions, obtaining joint running data for the operators. Finally, based on this joint running data, training performance analysis of the deep learning model is performed, yielding the training performance analysis results. Furthermore, a pipeline parallel strategy can be used to correlate the first running data after unifying the timeline, obtaining joint data and establishing a spatiotemporal correlation between the operator's running data in different dimensions. This establishes a connection between operator running data and the pipeline structure, providing a data foundation for model performance analysis and anomaly localization. When analyzing the joint running data, all related information can be clearly obtained, reducing the time spent on model training performance analysis and thus improving the efficiency of model training performance analysis. At the same time, the joint running data includes the first running data of the operator in different dimensions and the related information of each first running data. Performance analysis based on the joint running data can improve the accuracy of the model training performance analysis results.

[0041] Figure 4 This is a flowchart illustrating the steps of another deep learning model training performance analysis method provided in this application embodiment. For example... Figure 4 As shown, the method includes steps 401-410.

[0042] Step 401: While the deep learning model is being trained, obtain the first running data of the operators in the deep learning model in different dimensions; the deep learning model includes at least two operators.

[0043] The specific implementation method of step 401 can be referred to the above. Figure 2 Step 201 shown will not be repeated here.

[0044] Step 402: Determine the mapping information of the first running data of the operator in different dimensions through a preset association algorithm.

[0045] Step 403: Based on the mapping information, correlate the first running data of the operator in different dimensions to obtain the joint running data of the operator.

[0046] In this embodiment, for steps 402-403, the mapping information is used to characterize the correspondence between the first running data in different dimensions. The deep learning model may include at least two stages, each stage consisting of multiple consecutive model layers, and each stage may include at least two operators.

[0047] For example, taking model training via pipelined parallelism, based on the deployment configuration before model training, the pipelined parallelism strategy for model training can be obtained. According to this strategy, the mapping information between the first running data of the operator in the spatial dimension can be determined. Based on real-time acquired data, the mapping information between the first running data of the operator in the temporal dimension can be determined. Then, based on the mapping information between the first running data of the operator in the spatial dimension and the first running data of the operator in the temporal dimension, the mapping information between the first running data of the operator in both the spatial and temporal dimensions can be determined. Based on the mapping information between the first running data of the operator in the spatial and temporal dimensions, the first running data of the operator in different dimensions can be associated to obtain the joint running data of the operator.

[0048] In this embodiment, a preset association algorithm is used to determine the mapping information of the first running data of the operator in different dimensions. Then, based on the mapping information, the first running data of the operator in different dimensions are associated to obtain the joint running data of the operator. This can provide clear and reusable rules for data association, improve the objectivity of data association, thereby enhancing the reliability of data association results and further improving the accuracy of performance analysis based on joint running data.

[0049] Optionally, step 402 above may include sub-steps 4021-4024.

[0050] Sub-step 4021: Obtain the timestamps of the operators and micro-batch data, the stage corresponding to the operator, the information of the hardware devices in the processing stage, and the communication topology.

[0051] In this embodiment, the model includes at least two stages that can be processed by distributed hardware devices. This can also be understood as the model including at least two stages being deployed on multiple hardware devices, each responsible for processing its own deployed stage. The training data of the deep learning model is divided into multiple different micro-batch data, with dimensions including temporal and spatial dimensions. The communication topology is used to characterize the structural information of the communication links between stages. The information of the hardware devices processing the stages can be determined by the cluster topology. The cluster topology is used to characterize the structural information of all hardware devices (or nodes) within the distributed cluster. The timestamps of operators and micro-batch data can include start and end timestamps.

[0052] Sub-step 4022: Based on the timestamp of the operator and the timestamp of the micro-batch data, determine the first mapping information between the operator and the micro-batch data with the same timestamp.

[0053] For example, the timestamp of operator 1 includes a start timestamp and an end timestamp, with the start timestamp being 10 milliseconds (ms) and the end timestamp being 20 ms. The timestamp of micro-batch data 1 also includes a start timestamp and an end timestamp, with the start timestamp being 10 ms and the end timestamp being 20 ms. Since the timestamp of operator 1 is the same as the timestamp of micro-batch data 1, the first mapping information between operator 1 and micro-batch data 1 can be determined.

[0054] For example, an operator is the basic computational unit for model training and inference. A micro-batch of data may be computed by multiple operators. In this case, the start timestamp of operator 2 is 50 ms and the end timestamp is 55 ms. The start timestamp of micro-batch data 2 is 40 ms and the end timestamp is 60 ms. Then the running time of operator 2 is within the time window of micro-batch data 2, and the first mapping information between operator 2 and micro-batch data 2 can be determined.

[0055] Sub-step 4023: Based on the information of the hardware devices of the corresponding stage and processing stage and the communication topology, determine the second mapping information of the distributed hardware devices of the operator, stage, and processing stage and the communication links between stages.

[0056] For example, taking model training via pipelined parallelism as an example, based on the deployment configuration before model training, the pipelined parallelism strategy for model training can be obtained. This pipelined parallelism strategy may include the stage corresponding to the operator, the information of the hardware device of the processing stage, and the communication topology. For example, the pipelined parallelism strategy may include the model including stage 1, stage 2, and stage 3. Stage 1 is processed by hardware device 1, stage 2 is processed by hardware device 2, and stage 3 is processed by hardware device 3. Stage 1 includes operator 1 and operator 2, stage 2 includes operator 3, operator 4, and operator 5, and stage 3 includes operator 6 and operator 7. Stage 1 and stage 2 communicate through communication link 1, and stage 2 and stage 3 communicate through communication link 2. Based on the information contained in the above pipelined parallelism strategy, the second mapping information between operator 1, stage 1, hardware device 1, and communication link 1 can be determined. Similarly, the second mapping information between operator 2, stage 1, hardware device 1, and communication link 1 can also be determined.

[0057] Sub-step 4024: Based on the first mapping information and the second mapping information, determine the mapping information of the first running data of the operator in different dimensions.

[0058] In this embodiment of the application, the mapping information between the first running data of the operator in the time dimension and the space dimension can be determined based on the first mapping information between the first running data of the operator in the time dimension and the second mapping information between the first running data of the operator in the space dimension.

[0059] Based on the above embodiments, by way of example, after determining the first mapping information between operator 1 and micro-batch data 1, and the second mapping information between operator 1, stage 1, hardware device 1 and communication link 1, the mapping information between operator 1, micro-batch data 1, stage 1, hardware device 1 and communication link 1 can be determined, thereby obtaining the mapping information between the first running data of the operator in the time dimension and the spatial dimension.

[0060] This embodiment first obtains the timestamps of the operator and micro-batch data, the stage corresponding to the operator, the hardware device information of the processing stage, and the communication topology. Then, based on the timestamps of the operator and the micro-batch data, it determines the first mapping information between operators and micro-batch data with the same timestamp. By judging whether the time windows formed by the timestamps overlap, it accurately binds the operator and the micro-batch data. Based on the stage corresponding to the operator, the hardware device of the processing stage, and the communication topology, it determines the second mapping information of the operator, the stage, the distributed hardware devices of the processing stage, and the communication links between stages. This allows us to obtain the mapping relationship between the operator's running data in the spatial dimension. Finally, based on the first and second mapping information, it determines the mapping information of the operator's first running data in different dimensions. This solves the problem of the separation between the time and spatial dimensions, constructs a complete running scenario for the operator, improves the accuracy of data association, and ensures that the data association of the operator covers all dimensions, thus ensuring the completeness of data association.

[0061] Optionally, after step 403, steps 404-405 can be performed.

[0062] Step 404: Calculate the preset training performance index values ​​for the deep learning model based on the joint operation data of the operators.

[0063] Step 405: Add the values ​​of the training performance metrics of the deep learning model to the training performance analysis results of the deep learning model.

[0064] In this embodiment of the application, for steps 404-405, the preset training performance indicators for the deep learning model may include one or more of the following: vacuolation rate, utilization rate, effective computation time, device idle time, scheduling wait time, communication time, and communication masking rate. Here, vacuolation can represent a time segment during pipelined parallel execution where certain computing devices are in an idle waiting state due to task dependencies or discontinuous scheduling.

[0065] For example, utilization can represent the utilization rate of all hardware devices involved in the computation during model training. When the utilization rate is calculated to be m, the training performance of the deep learning model can be characterized by the utilization rate m. For example, the value of m is positively correlated with the training performance of the model, and the utilization rate m can be added to the training performance analysis results of the deep learning model.

[0066] This embodiment calculates preset training performance metrics for the deep learning model based on the joint operation data of the operators. These metrics are then added to the training performance analysis results of the deep learning model. The training performance metrics can quantify the model's training performance in a pipelined parallel approach, enabling fine-grained evaluation of the model's training performance.

[0067] Optionally, the above sub-step 404 may include sub-steps 4041-4042.

[0068] Sub-step 4041: In the joint operation data of the operator, determine the parameters and parameter values ​​required to calculate the index value of the training performance index.

[0069] Sub-step 4042: Calculate the value of the training performance index based on the parameters, the parameter values, and the preset index calculation formula.

[0070] In this embodiment of the application, for sub-steps 4041-4042, when calculating the index value of the preset training performance index for the deep learning model, the parameters required to calculate the index value of the preset training performance index can first be determined, then the parameter values ​​of these parameters can be obtained, and then the obtained parameter values ​​can be substituted into the preset index calculation formula corresponding to the preset training performance index to calculate the index value of the preset training performance index.

[0071] For example, when calculating the voiding rate, the parameters required for calculation can be determined from the joint operation data of the operators: device wait time and total execution time. Device wait time is the idle time of the device, and total execution time is the sum of the device's running time during the complete pipeline parallel training process. If the device wait time is 10ms and the total execution time is 100ms, and the preset formula for calculating the voiding rate is the ratio of wait time to total execution time multiplied by 100%, then substituting the values ​​of wait time and total execution time into the preset formula for calculating the voiding rate yields a voiding rate of 10%.

[0072] When calculating utilization, the parameters needed to determine the utilization rate are effective computation time and total time, which can be obtained from the joint operation data of the operators. Effective computation time can be understood as the time the operator actually uses for core computational tasks, i.e., the time spent on non-computational tasks (such as waiting for data, idle time, redundant operations, etc.) after removing non-computational time from the total time, retaining only the time directly related to the core computations of model training; total time is the total duration of the pipeline from start to finish. If the effective computation time is 80ms and the total time is 110ms, and the preset utilization rate calculation formula is the ratio of effective computation time to total time multiplied by 100%, then substituting the values ​​of effective computation time and total time into the preset utilization rate calculation formula yields a utilization rate of 72.7%.

[0073] This embodiment determines the parameters and their values ​​required for calculating training performance metrics from the joint operation data of the operators. Then, based on the parameters, their values, and the preset metric calculation formula, the metric values ​​are calculated. This ensures that the parameters required for calculating each metric are based on the metric definition and formula derivation, and are extracted from the joint data, providing data support for metric value calculation, improving the efficiency and accuracy of metric value calculation, and simultaneously enhancing the adaptability of the metric to the model training scenario.

[0074] Optionally, after step 403, steps 406-407 may also be performed.

[0075] Step 406: Determine the running efficiency of the deep learning model based on the index values ​​of the training performance indicators and the preset running efficiency calculation formula.

[0076] Step 407: Add the running efficiency of the deep learning model to the training performance analysis results of the deep learning model.

[0077] In this embodiment of the application, for steps 406-407, the preset operating efficiency calculation formula can be expressed by formula (1). Formula (1) In formula (1), Ep can represent the running efficiency of the deep learning model; Teff can represent the effective computation time of the deep learning model; T idle T can represent the idle time of a device running a deep learning model; wait T can represent the scheduling wait time during the training process of a deep learning model; comm This can represent the communication time during the training process of a deep learning model. The running efficiency of the deep learning model can be calculated according to formula (1).

[0078] This embodiment determines the operational efficiency of a deep learning model based on the values ​​of training performance metrics and a preset operational efficiency calculation formula. Then, the operational efficiency of the deep learning model is added to the training performance analysis results. The operational efficiency of the model can be used to intuitively quantify the training performance of the model in a pipelined parallel approach. Furthermore, by calculating the operational efficiency of the model using multiple different training metrics, the limitations of evaluating model performance with a single metric are reduced, and the comprehensiveness of the assessment of the model's training performance in a pipelined parallel approach is improved.

[0079] Optionally, after step 403, steps 408-410 may also be performed.

[0080] Step 408: Obtain the simulated running data of the operator on the distributed hardware device.

[0081] In this embodiment, the deep learning model includes at least two stages, each stage including at least two operators. These two stages can be processed by distributed hardware devices. Simulated runtime data can be understood as theoretically expected performance data derived from the runtime data of operators on a single hardware device (e.g., the expected execution time and memory usage of operators in a distributed environment). That is, without actually deploying a distributed environment, the ideal performance of operators is predicted through a theoretical model, providing a benchmark for subsequent comparison with actual data.

[0082] Step 409: Compare the simulated running data and the joint running data of the operator to obtain the comparison results of the simulated running data and the joint running data.

[0083] Step 410: Add the comparison results to the training performance analysis results of the deep learning model.

[0084] In this embodiment of the application, regarding steps 409-410, based on the above embodiments, it is known that the joint running data is the operator-level performance data actually collected in the model training process in the distributed environment. The simulated running data of the operators and the joint running data can be compared, and the training performance of the model can be quantified by comparing the two results, so as to quantitatively evaluate the training performance of the model in the pipelined parallel mode.

[0085] Figure 5 This is a schematic diagram comparing simulated operation data and combined operation data provided in an embodiment of this application. For example... Figure 5 As shown, theoretical modeling represents simulated operational data, while actual observations represent joint operational data; rank i can represent one process, rank i+1 can represent another process, and the numbers in the figure represent the micro-batch data numbers. In theoretical modeling, the forward computation (f) process and the backward computation (b) process are closely linked, f + b represents the total time of the forward and backward computations, and the total time of the forward and backward computations of the two processes is perfectly aligned, with no additional time overhead. In actual observation, there are communication (comm) processes and equipment waiting processes between the forward and backward computation processes (i.e.,... Figure 5 (The blank space in the middle). The actual total time for rank i is f1 + b1 + comm1; the actual total time for rank i+1 is f2 + b2 + comm2. There is a difference between the total time for forward and backward calculations of the two processes, resulting in additional time loss. The additional time loss of the actual observation process can be calculated using the expression overhead = (f2 - f1) + (b2 - b1) + (comm2 - comm1).

[0086] This embodiment acquires simulated operation data of the operator on distributed hardware devices, compares the simulated operation data with the joint operation data, and obtains the comparison results. Finally, the comparison results are added to the training performance analysis results of the deep learning model. This method of quantifying model training performance by comparing simulated operator operation data with real-time operation data provides a different approach than evaluating model performance through runtime efficiency, improving the accuracy and comprehensiveness of model performance evaluation.

[0087] Optionally, step 408 may include sub-steps 4081-4083.

[0088] Sub-step 4081: Obtain the configuration information of the deep learning model.

[0089] In this embodiment of the application, the configuration information of the deep learning model includes the structural information of the deep learning model and the information of the single hardware device running the deep learning model. For example, the configuration information of the model may include the hierarchical structure of the model, such as the attention (Transformer) layer, the operator type (e.g., MatMul operator, Conv operator, etc.) and the hardware device information (e.g., GPU model, memory capacity, etc.).

[0090] Sub-step 4082: Based on the configuration information of the deep learning model, determine the second running data of the deep learning model's operators when running on a single hardware device.

[0091] In this embodiment, a benchmarking tool (such as PyTorch Profiler or a custom script) can be used to run a sub-module of the model on a single hardware device to collect operator-level runtime data, that is, runtime data of the operator running on a single hardware device.

[0092] Sub-step 4083: Based on the second running data, determine the simulated running data of the operator on the distributed hardware device.

[0093] In this embodiment, due to factors such as hardware device heterogeneity and network bandwidth, the second running data of the operator running on a single hardware device differs from the running data of the operator running on a distributed hardware device. The execution time of the operator on a distributed hardware device can be simulated based on historical data and the second running data of the operator running on a single hardware device. For example, if a similar operator has been trained in a similar distributed environment before, the historical time ratio can be directly reused. For instance, when the device models are the same, the computation time of the operator on a distributed hardware device is n times that on a single hardware device; the computation time of the operator on a distributed hardware device can be simulated and calculated using the computation time of the operator on a single hardware device. Alternatively, the execution time of the operator on a distributed hardware device can be simulated using a simulation model (e.g., a machine learning model) based on the second running data of the operator running on a single hardware device. For example, the data of the operator running on a single hardware device, along with data on hardware device heterogeneity and network bandwidth, can be input into a machine learning model, and the machine learning model can output the running data of the operator on a distributed hardware device.

[0094] This embodiment obtains the configuration information of the deep learning model, determines the second running data of the deep learning model's operators on a single hardware device based on the configuration information, and then determines the predicted running data of the operators on distributed hardware devices based on the second running data. This provides a data foundation for comparing the simulated running data and joint running data of the operators, and can predict potential performance bottlenecks of the model before model training begins.

[0095] Optionally, the above method may also include step 411-sub-step 413.

[0096] Step 411: Based on the indicator values ​​and preset ranges of the training performance indicators of the deep learning model, determine the indicator value of the first indicator that exceeds the preset range.

[0097] In this embodiment, the preset range represents a performance acceptable interval set based on business needs or experience, such as the upper and lower limits of training indicators like vacuolation rate and utilization rate. A first indicator exceeding the preset range can be understood as an abnormal first indicator. For example, if the preset range for vacuolation rate is 0-15%, then when the vacuolation rate is greater than 15% during model training, it is determined that the vacuolation rate exceeds the preset range, and the vacuolation rate becomes the first indicator.

[0098] Step 412: Based on the joint operation data of the operators, perform attribution analysis on the index value of the first index to obtain the attribution results corresponding to the index value of the first index.

[0099] In this application embodiment, based on the joint operation data of the operators, the root cause of the first indicator value exceeding the range can be traced. For example, the throughput is low due to excessive time consumption of a certain operator, or the cavitation rate is high due to blockage of a certain communication link.

[0100] Step 413: Based on the attribution results and preset optimization configuration information, determine the optimization strategy for the first indicator in order to optimize the training strategy of the deep learning model.

[0101] In this embodiment of the application, the preset optimization configuration information may include solutions for abnormal indicators based on historical data and experience. Figure 3 The training performance analysis system for the deep learning model shown may also include a dynamic tuning module. This module can identify the cause of any abnormal metrics, determine a solution to the abnormality, and optimize the training strategy of the deep learning model based on the solution. For example, based on the above embodiment, when the empty packet rate is greater than 15%, the corresponding optimization strategy in the preset optimization configuration information is to increase the data volume of each micro-batch or to divide the optimization stages, with the expected optimization result being a reduction in hardware idle time. When a certain stage experiences a computational bottleneck, the corresponding optimization strategy in the preset optimization configuration information is to reallocate operators to faster hardware devices, with the expected optimization result being a balance of load across stages. When communication latency is too high, the corresponding optimization strategy in the preset optimization configuration information is to enable gradient compression or overlapped communication and computation, with the expected optimization result being a reduction in data transmission time.

[0102] This embodiment determines the value of a first metric that exceeds the preset range based on the training performance metrics of the deep learning model and a preset range. Then, based on the joint operation data of the operators, it performs attribution analysis on the value of the first metric to obtain the corresponding attribution result. This allows for inferring specific operator or hardware issues from metric anomalies, avoiding getting stuck on the surface phenomena of metric anomalies. Finally, based on the attribution result and preset optimization configuration information, it determines an optimization strategy for the first metric to optimize the training strategy of the deep learning model. This identifies the specific problems causing different metric anomalies, and targeted optimization strategies are determined based on the specific causes, improving the success rate and accuracy of optimization. It supports continuous iterative optimization, and users can interactively adjust parameters (such as adding buffers). The system re-monitors and verifies the effect until the training efficiency reaches the target, thereby improving the training performance of the model.

[0103] Optionally, the above method may also include step 414-sub-step 416.

[0104] Step 414: Generate performance alert information based on the first indicator.

[0105] Step 415: Based on the performance alarm information, determine the device information of the first indicator on the distributed hardware device.

[0106] In this embodiment, for steps 414-415, the performance alarm information can be displayed as text or broadcast via voice. The performance alarm information may include the name of the training metric, the value of the training metric, and the preset range corresponding to the training metric. When a first metric exceeds the preset range, the hardware device information corresponding to the first metric can be determined based on the performance alarm information and the joint operation data, thus locating the metric anomaly.

[0107] Step 416: Mark the region corresponding to the first indicator on the visualization interface corresponding to the training performance indicators of the deep learning model.

[0108] In the embodiments of this application, Figure 3 The training performance analysis system for the deep learning model shown can also include a visualization module. This module can display the aforementioned training performance metrics. For example, a Gantt chart can be used to display the timeline of each stage in the pipelined parallel approach, comparing the estimated runtime with the actual runtime; a spatiotemporal graph uses time on the horizontal axis and hardware devices or stages on the vertical axis, using colors to distinguish computation, communication, and idle states; a heatmap can display the time consumed by different hardware devices in computation and communication operators; and a pie chart can display the percentage of computation time, the percentage of communication time, and the time breakdown of a stage. When a first metric exceeds a preset range, the corresponding area on the visualization interface can be marked, for example, by highlighting it or using a bounding box.

[0109] This application embodiment generates performance alert information based on a first indicator, determines the device information of the first indicator on distributed hardware devices based on the performance alert information, and marks the area corresponding to the first indicator on the visualization interface corresponding to the training performance indicator of the deep learning model. This allows for real-time triggering of performance alerts, timely identification and response to dynamically occurring performance problems, avoidance of training losses caused by abnormal delays, and improved efficiency in anomaly localization by marking abnormal areas on the visualization interface.

[0110] Figure 6 This is a schematic diagram illustrating the specific implementation process of a deep learning model training performance analysis method provided in this application embodiment, such as... Figure 6 As shown, in a specific embodiment, S1-S6 can be executed sequentially to achieve training performance analysis, visualization observation, and adjustment optimization of the deep learning model.

[0111] In summary, this application's embodiments firstly acquire the first running data of operators in the deep learning model across different dimensions during deep learning model training. This allows for the collection of operator-level performance data, refining the granularity of performance analysis data and improving the accuracy of subsequent performance analysis results. Simultaneously, real-time data acquisition during model training ensures that the collected data reflects the model's true training state, enabling timely identification and response to dynamically emerging performance issues and avoiding analytical biases caused by data lag. For example, an offline data acquisition and analysis approach involves exporting logs after model training. If the GPU experiences a sudden 10-millisecond load peak, the log might not capture this data due to the sampling interval, leading to biases in the model training performance analysis. Secondly, a pre-defined association algorithm is used to correlate the first running data of operators across different dimensions, obtaining joint running data for the operators. Finally, based on this joint running data, training performance analysis of the deep learning model is performed to obtain the training performance analysis results. By using a pipelined parallel strategy to correlate the first running data after unifying the timeline, joint data can be obtained. This establishes spatiotemporal correlations between the running data of each operator in different dimensions, realizing the connection between operator execution and pipeline structure. This provides a data foundation for model performance analysis and anomaly localization. Analysis based on joint running data clearly reveals all correlation information, reducing the time spent on model training performance analysis and thus improving its efficiency. Furthermore, joint running data includes the first running data of operators in different dimensions and the correlation information between these first running data points. Performance analysis based on joint running data can improve the accuracy of model training performance analysis results.

[0112] This application also provides a device for analyzing the training performance of deep learning models. Figure 7 This is a structural block diagram of a deep learning model training performance analysis device provided in an embodiment of this application. Figure 7 As shown, the training performance analysis device 700 of the deep learning model may include: an acquisition module 701, an association module 702, and an analysis module 703.

[0113] The acquisition module 701 is used to acquire the first running data of the operators in the deep learning model in different dimensions when the deep learning model is being trained; the deep learning model includes at least two operators; The association module 702 is used to associate the first running data of the operator in different dimensions through a preset association algorithm to obtain the joint running data of the operator; Analysis module 703 is used to perform training performance analysis on the deep learning model based on the joint operation data, and obtain the training performance analysis results of the deep learning model.

[0114] Optionally, the deep learning model includes at least two stages, each stage consists of multiple consecutive model layers, and each stage includes at least two operators. The association module 702 is specifically used to determine the mapping information of the first running data of the operator in different dimensions through a preset association algorithm; the mapping information is used to characterize the correspondence between the first running data in different dimensions; and the first running data of the operator in different dimensions is associated according to the mapping information to obtain the joint running data of the operator.

[0115] Optionally, at least two stages are processed by distributed hardware devices. The training data of the deep learning model is divided into multiple different micro-batch data, with dimensions including time and space. The association module 702 is specifically used to obtain the timestamps of operators and micro-batch data, the stage corresponding to the operator, the information of the hardware devices of the processing stage, and the communication topology. The communication topology is used to characterize the structural information of the communication links between stages. Based on the timestamps of the operators and micro-batch data, a first mapping information between operators and micro-batch data with the same timestamp is determined. Based on the stage corresponding to the operator, the information of the hardware devices of the processing stage, and the communication topology, a second mapping information is determined for the operator, stage, distributed hardware devices of the processing stage, and communication links between stages. Based on the first and second mapping information, the mapping information of the first running data of the operator in different dimensions is determined.

[0116] Optionally, the analysis module 703 is specifically used to calculate the index values ​​of the preset training performance indicators for the deep learning model based on the joint operation data of the operators; the training performance indicators include one or more of the following: bubble rate, utilization rate, effective computation time, device idle time, scheduling waiting time, communication time, and communication masking rate; and add the index values ​​of the training performance indicators of the deep learning model to the training performance analysis results of the deep learning model.

[0117] Optionally, the analysis module 703 is specifically used to determine the parameters and parameter values ​​required to calculate the training performance index in the joint operation data of the operator; and to calculate the training performance index value based on the parameters, parameter values ​​and preset index calculation formula.

[0118] Optionally, such as Figure 7 As shown, the training performance analysis device 700 of the deep learning model may further include: a determination module 704 and a processing module 705.

[0119] The determination module 704 is used to determine the running efficiency of the deep learning model based on the indicator values ​​of the training performance metrics and the preset running efficiency calculation formula. The processing module 705 is used to add the running efficiency of the deep learning model to the training performance analysis results of the deep learning model.

[0120] Optionally, the deep learning model includes at least two stages, each stage includes at least two operators, and the at least two stages are processed by a distributed hardware device. The analysis module 703 is specifically used to acquire simulated running data of the operators on the distributed hardware device; compare the simulated running data and the joint running data of the operators to obtain the comparison results of the simulated running data and the joint running data; and add the comparison results to the training performance analysis results of the deep learning model.

[0121] Optionally, the analysis module 703 is specifically used to obtain the configuration information of the deep learning model; the configuration information of the deep learning model includes the structural information of the deep learning model and the device information of the single hardware device running the deep learning model; based on the configuration information of the deep learning model, the second running data of the deep learning model's operator running on the single hardware device is determined; based on the second running data, the simulated running data of the operator on the distributed hardware device is determined.

[0122] Optionally, the determining module 704 is further configured to determine the value of a first indicator that exceeds the preset range based on the indicator value and preset range of the training performance indicator of the deep learning model; the processing module 705 is further configured to perform attribution analysis on the indicator value of the first indicator based on the joint operation data of the operators, and obtain the attribution result corresponding to the indicator value of the first indicator; the determining module 704 is further configured to determine the optimization strategy for the first indicator based on the attribution result and preset optimization configuration information, so as to optimize the training strategy of the deep learning model.

[0123] Optionally, the processing module 705 is also configured to generate performance alarm information based on the first indicator; Based on the performance alert information, determine the device information of the first indicator on the distributed hardware device; mark the area corresponding to the first indicator on the visualization interface corresponding to the training performance indicators of the deep learning model.

[0124] This application also provides an electronic device, with reference to... Figure 8 , Figure 8 This is a structural block diagram of an electronic device for analyzing the training performance of a deep learning model, provided in an embodiment of this application. Figure 8 As shown, the electronic device includes: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used to store executable instructions, which cause the processor to execute the training performance analysis and prediction method of the deep learning model in the aforementioned embodiment.

[0125] The processor can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable devices, transistor logic devices, hardware components, or any combination thereof. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

[0126] The communication bus may include a path for transmitting information between the memory and the communication interface. The communication bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0127] This application also provides a non-transitory computer-readable storage medium, which, when the instructions in the storage medium are executed by the processor of an electronic device (server or terminal), enables the processor to perform... Figure 2 or Figure 4 The method for analyzing the training performance of the deep learning model shown.

[0128] This application also provides a computer program product, which includes a computer program loaded and executed by a processor of an electronic device (server or terminal) to implement the training performance analysis method for the deep learning model as described above.

[0129] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0130] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0132] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing terminal device to operate in a predictive manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0134] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0135] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0136] The above provides a detailed description of the training performance analysis method, apparatus, electronic device, and computer-readable storage medium for a deep learning model provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for analyzing the training performance of a deep learning model, characterized in that, The method includes: When the deep learning model is trained, first operational data of the operators in the deep learning model in different dimensions are obtained; the deep learning model includes at least two of the operators. By using a preset association algorithm, the first running data of the operator in different dimensions are associated to obtain the joint running data of the operator; Based on the joint operation data, the training performance of the deep learning model is analyzed to obtain the training performance analysis results of the deep learning model.

2. The method according to claim 1, characterized in that, The deep learning model includes at least two stages, each stage consists of multiple consecutive model layers, and each stage includes at least two operators. The joint operational data of the operators is obtained by associating the first operational data of the operators in different dimensions using a preset association algorithm, including: A preset association algorithm is used to determine the mapping information of the first running data of the operator in different dimensions; the mapping information is used to characterize the correspondence between the first running data in different dimensions. Based on the mapping information, the first running data of the operator in different dimensions are correlated to obtain the joint running data of the operator.

3. The method according to claim 2, characterized in that, The at least two stages are processed by distributed hardware devices, and the training data of the deep learning model is divided into multiple different micro-batch data, the dimensions of which include time dimension and spatial dimension; The step of determining the mapping information of the operator's first running data in different dimensions through a preset association algorithm includes: The timestamps of the operator and the micro-batch data, the stage corresponding to the operator, the information of the hardware device processing the stage, and the communication topology are obtained; the communication topology is used to characterize the structural information of the communication links between the stages; Based on the timestamp of the operator and the timestamp of the micro-batch data, determine the first mapping information between the operator and the micro-batch data with the same timestamp; Based on the information of the stage corresponding to the operator, the hardware device processing the stage, and the communication topology, determine the second mapping information of the operator, the stage, the distributed hardware device processing the stage, and the communication link between the stages; Based on the first mapping information and the second mapping information, the mapping information of the first running data of the operator in different dimensions is determined.

4. The method according to claim 1, characterized in that, The step of performing training performance analysis on the deep learning model based on the joint operation data to obtain the training performance analysis results of the deep learning model includes: Based on the joint operation data of the operators, the indicator values ​​of the preset training performance indicators for the deep learning model are calculated; the training performance indicators include one or more of the following: voiding rate, utilization rate, effective computation time, device idle time, scheduling waiting time, communication time, and communication masking rate. The values ​​of the training performance metrics of the deep learning model are added to the training performance analysis results of the deep learning model.

5. The method according to claim 4, characterized in that, The step of calculating the preset training performance index value for the deep learning model based on the joint operation data of the operators includes: In the joint operation data of the operator, determine the parameters required to calculate the index value of the training performance index and the parameter values ​​of the parameters; The training performance index value is calculated based on the parameters, the parameter values, and the preset index calculation formula.

6. The method according to claim 4, characterized in that, The method further includes: The operating efficiency of the deep learning model is determined based on the value of the training performance index and the preset operating efficiency calculation formula. The running efficiency of the deep learning model is added to the training performance analysis results of the deep learning model.

7. The method according to claim 1, characterized in that, The deep learning model includes at least two stages, each stage including at least two operators, and the at least two stages are processed by distributed hardware devices. The training performance analysis of the deep learning model is performed based on the jointly executed data to obtain the training performance analysis results of the deep learning model, including: Obtain the simulated operation data of the operator on the distributed hardware device; The simulated running data of the operator and the joint running data are compared to obtain the comparison results of the simulated running data and the joint running data; The comparison results are added to the training performance analysis results of the deep learning model.

8. The method according to claim 7, characterized in that, The step of obtaining the simulated operation data of the operator on the distributed hardware device includes: Obtain the configuration information of the deep learning model; the configuration information of the deep learning model includes the structural information of the deep learning model and the device information of the single hardware device running the deep learning model; Based on the configuration information of the deep learning model, second running data for the operators of the deep learning model are determined when they are running on the single hardware device. Based on the second running data, the simulated running data of the operator on the distributed hardware device is determined.

9. The method according to claim 4, characterized in that, The method further includes: Based on the indicator values ​​and preset ranges of the training performance indicators of the deep learning model, determine the indicator value of the first indicator that exceeds the preset range. Based on the joint operation data of the operators, attribution analysis is performed on the index value of the first index to obtain the attribution result corresponding to the index value of the first index. Based on the attribution results and preset optimization configuration information, an optimization strategy for the first indicator is determined to optimize the training strategy of the deep learning model.

10. The method according to claim 9, characterized in that, The method further includes: Based on the first indicator, generate performance alert information; Based on the performance alarm information, determine the device information of the first indicator on the distributed hardware device; The region corresponding to the first indicator on the visualization interface corresponding to the training performance indicator of the deep learning model is marked.

11. A device for analyzing the training performance of a deep learning model, characterized in that, The device includes: The acquisition module is used to acquire first running data of the operators in the deep learning model in different dimensions when the deep learning model is being trained; the deep learning model includes at least two of the operators. The association module is used to associate the first running data of the operator in different dimensions through a preset association algorithm to obtain the joint running data of the operator; The analysis module is used to perform training performance analysis on the deep learning model based on the joint operation data, and obtain the training performance analysis results of the deep learning model.

12. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as described in any one of claims 1 to 10.

Images