Performance simulation and closed-loop calibration method and system for distributed neural network training

Abstract

The application discloses a performance simulation and closed-loop calibration method and system for distributed neural network training; the method comprises the following steps: obtaining operator execution efficiency parameters of a calculation operator and a communication operator through benchmark measurement, and constructing a multi-dimensional characteristic performance database; extracting an operator execution feature combination, and obtaining a module-level performance prediction result through training performance simulation based on the multi-dimensional characteristic performance database; obtaining module-level real training performance data in a real training process, comparing the module-level real training performance data with the module-level performance prediction result, performing module-level error analysis, and obtaining a module-level error analysis result; calibrating and optimizing the benchmark measurement method or the performance simulation model structure according to the module-level error analysis result, and re-executing structured training performance simulation, so that a closed-loop optimization process among performance measurement, performance simulation, error analysis and calibration is formed; and the application can be used for distributed neural network training performance modeling, performance prediction, parallel strategy evaluation and system resource planning.

Description

Technical Field

[0001] This invention relates to the field of performance modeling and simulation technology for distributed deep learning systems, and in particular to a performance simulation and closed-loop calibration method and system for distributed neural network training. Background Technology

[0002] In the training of distributed neural networks, as the model size and parallelism scale continue to expand, accurate prediction of training time is crucial for system planning, hardware selection, and parallel strategy design. Since the performance of distributed training is typically determined by both the execution efficiency of computational operators and the efficiency of ensemble communication, existing technologies generally employ a "performance benchmarking + performance simulation" approach to predict performance before training: first, typical operators are benchmarked to obtain their performance parameters on the target hardware platform; then, during training simulation, a performance simulation model is built based on these parameters to estimate the overall training time.

[0003] However, in existing technologies, computing and communication are typically modeled separately, which has the following significant limitations: In terms of computation, existing methods often categorize operators by type, such as measuring the average execution efficiency of operators like matrix multiplication, attention computation, normalization, and activation functions, and assigning a uniform efficiency factor to each type of operator. However, in actual computation, the actual execution efficiency of operators is not only related to the operator type but also closely related to multidimensional features such as the shape of the input tensor, data arrangement, parallel partitioning strategy, memory access pattern, and hardware resource utilization. Existing techniques simplify operator performance to a single type dimension, ignoring multidimensional feature factors, which can easily lead to large deviations in efficiency estimation under different feature conditions, thus affecting the overall simulation accuracy.

[0004] In terms of communication, existing methods typically categorize communication based on the type of communication operator, such as all-reduce, all-gather, and all-to-all, and set fixed bandwidth or latency models for different types of communication operators. However, in actual distributed training, communication performance depends not only on the type of communication operator, but also on factors such as the amount of communication data, the size of the communication participation group, the intra-machine and inter-machine topology, link bandwidth, latency parameters, and the overlap between communication and computation. Coarse-grained modeling based solely on communication operator categories is insufficient to accurately reflect real-world communication behavior in complex parallel environments.

[0005] Furthermore, existing performance simulation processes typically only perform macroscopic comparisons and verifications of the overall training time after prediction is completed, lacking the ability to decompose errors at the module or sub-module level. When there are deviations between the simulation prediction results and the actual training results, it is difficult to determine whether the error originates from computational modeling, communication modeling, or specific module structural assumptions, thus making it difficult to provide an effective calibration basis for the performance prediction model.

[0006] Furthermore, in existing technologies, performance benchmark measurement, performance simulation, and real training performance analysis are usually independent of each other, lacking a systematic feedback mechanism. Once simulation parameters are set, they often remain unchanged for a long time, making it impossible to dynamically correct the modeling dimension, feature space, or simulation structure based on real training results, which severely limits the continuous optimization capability of performance prediction models.

[0007] Therefore, there is an urgent need for a closed-loop calibration system that can perform feature modeling in both computational and communication dimensions and drive the optimization of measurement strategies and simulation structures through a module-level error localization mechanism, so as to improve the accuracy and stability of distributed neural network training performance prediction. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of the prior art by providing a performance simulation and closed-loop calibration method and system for distributed neural network training, which solves the problems of single modeling dimension, insufficient error localization granularity, and lack of continuous calibration mechanism in the existing performance simulation model.

[0009] To achieve the above objectives, the present invention proposes the following technical solution: In a first aspect, the present invention provides a performance simulation and closed-loop calibration method for distributed neural network training, comprising the following steps: S1: Construct a multi-dimensional performance feature space for computational operators and communication operators, and generate operator execution feature combinations for computational operators and communication operators based on the multi-dimensional performance feature space; then perform benchmark measurements on these combinations to construct operator execution efficiency parameters; and finally, store the operator execution feature combinations and operator execution efficiency parameters in a structured manner to establish a multi-dimensional feature-based performance database. S2: Construct a module hierarchy based on the neural network model structure, extract operator execution feature combinations based on model structure information, training configuration parameters, and hardware deployment configuration; query the corresponding performance parameters in the multidimensional feature-based performance database, and predict the operator execution time based on the queried performance parameters; aggregate the operator execution time based on the module hierarchy and scheduling model to obtain module-level performance prediction results; S3: Obtain module-level real training performance data during the actual training process; establish the module correspondence between module-level real training performance data and module-level performance prediction results, compare the module-level real training performance data with the module-level performance prediction results, perform module-level error analysis, and obtain module-level error analysis results; S4: Based on the module-level error analysis results, the benchmark measurement method or performance simulation model is calibrated and optimized, and the structured training performance simulation is re-executed, thereby forming an iterative optimization closed loop between performance measurement, performance simulation, error analysis and calibration.

[0010] Further, in step S1, the multidimensional performance feature space is specifically a feature set used to describe the operating characteristics of the operator under different execution conditions. The multidimensional performance feature space can be predefined according to the operator type, training configuration parameters, parallel strategy, hardware platform information and communication environment information in the neural network model, or it can be automatically generated by the system after parsing the model structure and training configuration. The multidimensional performance feature space includes a computation operator feature space and a communication operator feature space, wherein, The computational operator feature space includes, but is not limited to, operator type, input tensor shape, data memory layout, and running hardware platform information; The communication operator feature space includes, but is not limited to, communication operator type, communication data volume, communication participant group size, and communication bandwidth and delay parameters.

[0011] Further, in step S1, the operator execution efficiency parameter is used to characterize the actual execution efficiency of the operator under a specific combination of operator execution characteristics, wherein, For a computation operator, the operator execution efficiency parameter represents the proportional relationship between the actual computation performance and the theoretical peak computation capability of the hardware; For a communication operator, the operator execution efficiency parameter represents the ratio between the effective bandwidth and the theoretical bandwidth during the communication process; The multidimensional characteristic performance database is obtained by structurally storing the operator execution efficiency parameters according to the operator execution feature combination; the multidimensional characteristic performance database is used to establish the mapping relationship between the operator execution feature combination and the operator execution efficiency parameters. The multidimensional characteristic performance database includes multiple data records, each data record corresponding to a set of operator execution feature combinations and their corresponding performance parameters.

[0012] Further, in step S2, the operator execution feature combination is obtained through an automatic parsing mechanism. The performance parameters are a set of data used to characterize the operator's performance under the corresponding operator execution feature combination. It includes at least the operator execution efficiency parameter; it also includes the statistical results of the original measurement data, stable performance measurement values, theoretical calculation volume, theoretical communication volume, measurement hardware environment identifier, and the version number of the characteristic performance database. The automatic parsing mechanism includes: parsing the model structure definition file to obtain the module hierarchy, operator type, and operator parameters; parsing the training configuration parameters to obtain the parallel strategy, computation task partitioning method, and communication group information; parsing the hardware deployment configuration to obtain the device type, number of devices, peak computing power of devices, communication topology, communication bandwidth, and latency parameters; and generating operator execution feature combinations for each operator based on the above parsing results during structured training performance simulation, and querying the corresponding operator execution efficiency parameters in the multidimensional feature-based performance database according to the operator execution feature combinations. The specific execution time of the operator is as follows: for a computation operator, the execution time of the operator is calculated based on the theoretical computational load, the operator execution efficiency parameters, and the peak computing power of the device; For communication operators, the operator execution time is calculated based on theoretical communication volume, operator execution efficiency parameters, theoretical bandwidth, and latency.

[0013] Further, in step S2, The aggregation of module execution time based on operator execution time specifically involves: during module-level time aggregation, a scheduling model is established based on the module hierarchy and operator execution dependencies. The system then performs bottom-up aggregation calculations on the execution time based on the module hierarchy to obtain the module-level performance prediction results. The scheduling model includes operator-level scheduling modeling and module-level scheduling modeling. In the operator-level scheduling modeling, the operator execution dependency includes at least one of the following: a first sequential execution relationship; a first parallel execution relationship; and a computation and communication overlapping execution relationship. In the module-level scheduling modeling, the module execution dependency relationship includes at least one of the following: a second sequential execution relationship; a second parallel execution relationship; and a pipelined parallel execution relationship. When the execution relationship is first or second sequential, the execution time is the sum of the execution times of the corresponding subtasks. When the execution relationship is first or second parallel, the execution time is the maximum value of the execution times of the parallel subtasks, or the sum of the maximum value and the synchronization overhead. When the execution relationship is computation and communication overlapping, the execution time is determined based on the computation time, communication time, and overlap ratio. When the execution relationship is pipelined, the overall execution time is determined based on the number of pipeline stages, the number of micro-batches, the execution time of each stage, and the scheduling strategy.

[0014] Furthermore, in step S3, the module-level real training performance data specifically refers to the module execution time data corresponding to the model module hierarchical structure, which is collected and organized during the real training process. The module-level error analysis establishes a module correspondence between module-level performance prediction results and module-level real training performance data based on the hierarchical structure of the neural network model modules. Error analysis is performed on the module-level real training performance data and module-level performance prediction results to obtain the errors of each module. The errors are then summarized layer by layer according to the model hierarchy to obtain the error hierarchy distribution results. Furthermore, the error hierarchy distribution results are classified and attributed based on the module's internal operator composition, module execution stage, and corresponding computational or communication characteristics to obtain error attribution results. Finally, the system generates module-level error analysis results based on the errors of each module, the error hierarchy distribution results, and the error attribution results. Specifically, the error hierarchy distribution result is obtained by first analyzing the error situation of the leaf nodes, and then summarizing it up to the intermediate nodes and the root node, so as to obtain the error distribution of different level modules, which is the error hierarchy distribution result. The module correspondence is established in the following way: (1) The module hierarchy structure generated in the performance simulation stage and the module hierarchy structure obtained in the real training stage are standardized to obtain a module hierarchy path in a unified format. (2) Match the predicted modules and the real modules according to the module name and module hierarchical path; (3) When the module names are completely identical and the module hierarchical paths are identical, establish an exact module correspondence; (4) When there are differences in frame prefix, numbering or naming format in the module name, the module name shall be standardized before the corresponding relationship is established; (5) When a prediction module corresponds to multiple real sub-modules, or multiple prediction modules correspond to one real module, an aggregate correspondence relationship is established based on the module hierarchical path and the parent-child node relationship; The error includes: (1) Absolute error, which is the absolute value of the difference between the module execution time collected in the actual training and the module execution time predicted in the simulation; (2) Relative error, which is the ratio of absolute error to the actual module execution time collected during training; (3) Error contribution, which is used to characterize the degree of influence of the module error on the overall training time error; The error attribution results include: (1) Calculate the deviation of operator performance parameters; (2) Deviation in the performance parameters of the communication operator; (3) The modeling of module execution scheduling relationships is inaccurate; (4) No operators were modeled, and execution behavior was not covered; (5) Insufficient feature space modeling.

[0015] Further, in step S4, the model calibration optimization includes adjusting the benchmark measurement method and correcting the performance simulation model structure; wherein, the benchmark measurement method adjustment includes, but is not limited to: expanding the test parameter range of the operator; increasing the sampling density of the feature region; and increasing the feature dimension of the operator; The performance simulation model structure modification includes modifying the performance simulation model structure, including but not limited to: supplementing the performance model with missing operators; and modifying the operator scheduling modeling of the simulation model.

[0016] Secondly, the present invention provides a performance simulation and closed-loop calibration system for distributed neural network training, comprising: The performance feature construction unit is used to construct a multi-dimensional feature space for computational and communication operators, and to obtain operator execution efficiency parameters through benchmark measurements to form a performance database. The performance simulation prediction unit is used to perform performance simulation prediction based on the neural network model structure and training configuration, and output module-level performance prediction results. The module-level error analysis unit is used to acquire module-level real training performance data during the actual training process, compare it with the module-level performance prediction results, perform module-level error decomposition, and obtain module-level error analysis results. The model calibration and optimization unit is used to adjust the performance database or performance simulation model based on the module-level error analysis results, and re-execute the performance simulation prediction to achieve closed-loop optimization of the performance model.

[0017] Thirdly, the present invention provides an electronic device, including a memory and a processor, characterized in that the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the performance simulation and closed-loop calibration method for distributed neural network training described above.

[0018] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the performance simulation and closed-loop calibration method for distributed neural network training.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention constructs a multi-dimensional performance feature space for computational and communication operators, incorporating factors such as operator type, tensor shape, data layout, communication data volume, communication group size, hardware platform, and network parameters into the performance modeling process. Compared to configuring uniform efficiency parameters only according to operator type, this invention can more accurately characterize the differences in operator performance under different execution feature conditions and improve the accuracy of performance prediction. 2. This invention uses a multi-dimensional characteristic performance database to structurally store the combination of operator execution features and operator execution efficiency parameters, enabling the performance simulation process to automatically query the corresponding performance parameters according to the specific training task and hardware deployment conditions, thereby improving the reusability of performance parameters and the adaptability of the model; 3. This invention outputs module-level performance prediction results and compares them with actual module-level training performance data during the real training process. By establishing a module correspondence between the real module and the predicted module, the overall performance error can be decomposed into specific model modules or sub-modules, thereby identifying the modules, hierarchical positions, and error contributions within the error set, avoiding the problem of difficulty in locating the source of error by relying solely on overall training time comparison. Furthermore, during module-level error analysis, the composition of operators within the module, the combination of operator execution characteristics, the computational communication ratio, and scheduling relationships can be combined to classify and attribute the sources of error. 4. Based on module-level error analysis results, this invention adjusts the benchmark measurement method, multi-dimensional characteristic performance database, and performance simulation model. For example, it can expand the range of operator test parameters, increase the sampling density of feature regions, add new operator execution feature dimensions, supplement unmodeled operators, or correct internal scheduling relationships within modules. Through these methods, an iterative optimization closed loop is formed between performance measurement, performance simulation, error analysis, and model correction, enabling the performance model to be continuously updated and optimized based on real training results. 5. This invention can predict training performance under different model structures, parallel strategies, hardware deployment methods, and communication configurations before actual training, and continuously calibrate the performance model using real training data. Therefore, this invention can be used for hardware selection, resource planning, parallel strategy design, performance bottleneck analysis, and system optimization evaluation in large-scale distributed neural network training. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is the overall flowchart of the performance simulation and closed-loop calibration system of the present invention; Figure 2 This is a flowchart of step S1 of the present invention, which involves constructing a multidimensional characteristic performance database. Figure 3 This is a flowchart of the structured training performance simulation in step S2 of the present invention; Figure 4 This is a flowchart of the module-level error analysis step S3 of the present invention; Figure 5 This is a flowchart of step S4 of the present invention, which involves performing calibration optimization based on error analysis. Figure 6 This is a framework diagram of the performance simulation and closed-loop calibration system of the present invention; Figure 7 This is a structural diagram of the electronic device provided by the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0024] Example 1: A Performance Simulation and Closed-Loop Calibration Method for Distributed Neural Network Training Firstly, this invention provides a performance simulation and closed-loop calibration method for distributed neural network training, the flowchart of which is shown below. Figure 1 As shown, this method combines performance benchmark measurement, performance simulation prediction, and module-level error analysis to construct an iterative optimization process of measurement-simulation-error localization-modeling correction, thereby enabling the performance model to continuously approximate the real training behavior.

[0025] S1. Construct a multidimensional characteristic performance database like Figure 2 As shown, this invention first constructs a multi-dimensional performance feature space for computational and communication operators, and then generates different combinations of operator execution features based on this multi-dimensional performance feature space. Benchmark measurements are then performed on these combinations to construct corresponding operator execution efficiency parameters. The combinations of operator execution features and the operator execution efficiency parameters are then stored in a structured manner to establish a multi-dimensional characteristic performance database. This multi-dimensional characteristic performance database stores operator execution efficiency parameters under different feature combinations, providing a queryable performance parameter foundation for subsequent performance simulations.

[0026] S1.1 Constructing the multidimensional performance feature space of the operator For computational operators and communication operators, multi-dimensional performance feature spaces (computational operator feature space and communication operator feature space) are constructed separately to describe the operational characteristics of operators under different execution environments. The multi-dimensional performance feature space refers to the set of features used to describe the operational characteristics of operators under different execution conditions. The multi-dimensional performance feature space can be predefined based on the operator type, training configuration parameters, parallelism strategy, hardware platform information, and communication environment information in the neural network model, or it can be automatically generated by the system after parsing the model structure and training configuration.

[0027] For computation operators, the feature space of the computation operator may include, but is not limited to, the following features: (1) Calculate the operator type; (2) Input the tensor shape; (3) Data memory layout; (4) Running hardware platform information.

[0028] For a communication operator, the feature space of the communication operator may include, but is not limited to, the following features: (1) Communication operator types; (2) Communication data volume; (3) Size of the communication participation group; (4) Bandwidth and delay parameters.

[0029] By introducing a multidimensional performance feature space, the problem of insufficient accuracy caused by performance modeling based solely on operator type can be avoided, thereby improving the granularity of modeling.

[0030] S1.2 Obtaining Operator Execution Efficiency Parameters Based on Benchmark Measurements After constructing a multidimensional performance feature space, benchmark measurements are performed on different combinations of operator execution features of computational and communication operators.

[0031] The benchmark measurement refers to the process of conducting actual operational tests on computational and communication operators under specific execution characteristic conditions on a target hardware platform to obtain their true execution performance data. The benchmark measurement can be completed by constructing operator test programs or calling existing operator implementation interfaces, executing multiple sets of test tasks in a controlled environment.

[0032] Reference measurements are performed separately for the computation operator and the communication operator, as follows: Calculation operator reference measurement process: (1) Generate a corresponding computation operator execution configuration based on the feature combination, wherein the computation operator execution configuration includes, but is not limited to, input tensor shape, data memory layout, computation precision and running device information; (2) Run the computation operator multiple times on the target hardware platform and collect the results of the first multiple runs; the results of the first multiple runs include actual execution time or throughput performance data; (3) Perform statistical processing on the results of the first multiple runs to eliminate random fluctuations and obtain stable computational performance measurements.

[0033] The statistical processing may include at least one of the following: removing the results from the preheating phase; calculating the average or median of the first multiple runs; removing the maximum, minimum, or abnormal fluctuation values; calculating the standard deviation or coefficient of variation, and re-performing the baseline measurement when the fluctuation exceeds a preset threshold. Through the above statistical processing, the impact of accidental operational fluctuations on the measurement results can be reduced.

[0034] Communication operator reference measurement process: (1) Generate a corresponding communication operator execution configuration based on the combination of features. The communication operator execution configuration includes, but is not limited to, communication operator type, communication data volume, number of communication participating devices, communication topology, and bandwidth and delay parameters. (2) Execute the communication operator multiple times in a distributed operating environment and collect the results of the second multiple executions, which include communication time or effective bandwidth data; (3) Perform statistical processing on the second multiple run results to eliminate random fluctuations and obtain stable communication performance measurement values.

[0035] The statistical processing may include at least one of the following: removing the results from the preheating phase; calculating the average or median of the second or multiple runs; removing the maximum, minimum, or abnormal fluctuation values; calculating the standard deviation or coefficient of variation; and re-performing the baseline measurement when the fluctuation exceeds a preset threshold. Through the above statistical processing, the impact of accidental operational fluctuations on the measurement results can be reduced.

[0036] The computational performance measurement value and the communication performance measurement value can be obtained based on the benchmark measurement, and these are defined as benchmark test results; the operator execution efficiency parameters are constructed based on the benchmark measurement results.

[0037] Specifically, the operator execution efficiency parameter is used to characterize the actual execution efficiency of the operator under specific combinations of execution characteristics, and its definition is as follows: For computation operators, the operator execution efficiency parameter can be used to represent the ratio between the actual computing power and the peak theoretical computing power of the hardware platform; for communication operators, the operator execution efficiency parameter can be used to represent the ratio between the effective bandwidth and the theoretical bandwidth of the communication process.

[0038] Furthermore, in order to estimate the execution time based on the operator's execution efficiency parameters, it is necessary to define the theoretical computational cost and theoretical communication cost of the operator.

[0039] The theoretical computational cost of an operator refers to the total number of computational operations determined by its mathematical definition, without considering hardware implementation efficiency. It characterizes the computational complexity of the operator and can be measured in floating-point operations (FLOPs). For example, the theoretical computational cost of a matrix multiplication operator can be calculated based on the matrix dimension; for a convolution operator, it can be determined based on the kernel size, input / output feature map dimensions, and number of channels. For other types of operators, their theoretical computational cost can also be determined based on their corresponding mathematical operation forms.

[0040] The theoretical communication volume of the operator refers to the total amount of data that the communication operator needs to transmit during the execution of distributed training. It is used to characterize the scale of communication overhead and can be measured in bytes. For example, for the all-reduce operator, the theoretical communication volume can be calculated based on the number of participating devices and the size of the data to be reduced.

[0041] By calculating the theoretical computational and communication costs of the operator and combining this with the corresponding operator execution efficiency parameters, the execution time of the operator under specific combinations of execution features can be calculated, thereby supporting quantitative prediction of training performance.

[0042] For computational operators, the execution time can be estimated by combining the theoretical computational load, the peak computing power of the device, and the operator's execution efficiency parameters. If the theoretical computational load of the operator is F, the theoretical peak computing power of the target hardware platform is P, and the corresponding operator execution efficiency parameter is , then the execution time of the computational operator can be estimated using the following formula: For communication operators, the communication time can be estimated based on the theoretical communication volume, theoretical bandwidth, and delay parameters, combined with the operator execution efficiency parameters. If the theoretical communication volume of the operator is B, the theoretical communication bandwidth is W, the corresponding operator execution efficiency parameter is [value missing], and the communication delay parameter is L, then the execution time of the communication operator can be estimated using the following formula: S1.3 Constructing a Multidimensional Characteristic Performance Database After obtaining the operator execution efficiency parameters, the system stores the operator execution efficiency parameters in a structured manner according to the combination of operator execution features, and constructs a multi-dimensional feature-based performance database.

[0043] The multidimensional characteristic performance database is used to establish a mapping relationship between operator execution feature combinations and operator execution efficiency parameters. Specifically, the multidimensional characteristic performance database includes multiple data records, each data record corresponding to a set of operator execution feature combinations and their corresponding performance parameters. Each data record includes at least the following: (1) Operator identification information: used to identify the type of operator, such as computation operator type or communication operator type; (2) Execution characteristic information: used to describe the combination of characteristics of the operator in a specific execution environment, including but not limited to the shape of the input tensor, data memory layout, parallel strategy, running hardware platform information (for computation operators), as well as the amount of communication data, the scale of communication participating devices, communication topology, bandwidth and delay parameters (for communication operators). (3) Performance parameters: A set of data used to characterize the performance of an operator under the corresponding combination of operator execution features, which includes at least the operator execution efficiency parameter; optionally, it also includes the statistical results of the original measurement data, stable performance measurement values, theoretical computational load, theoretical communication load, measurement hardware environment identifier, and the version number of the characteristic performance database. Therefore, the operator execution efficiency parameter is the core parameter among the performance parameters, and the range of the performance parameters is larger than that of the operator execution efficiency parameter; (4) Auxiliary information: including measurement time, measurement environment identifier, data version number, etc., to support performance data management and updates.

[0044] In one implementation, specifically, the multidimensional characteristic performance database is indexed according to combinations of operator execution features. The performance database supports: (1) Feature combination query: The performance parameters corresponding to the feature combination query are executed according to the input operator. When the feature combination of the queried operator does not completely match the existing records in the performance database, the performance database also supports approximate query or interpolation calculation based on feature similarity to obtain the estimated performance parameters. (2) Adding new feature combinations: When performance parameters under new operator execution feature combinations (i.e., new value combinations under existing feature dimensions) are obtained through benchmark measurement, corresponding data records can be added to the performance database; (3) Feature space expansion: When new operator execution feature dimensions are introduced during performance modeling, the performance database structure can be expanded to support the storage of performance parameters under the new feature dimensions; (4) Performance parameter update: When the benchmark measurement results are updated, the existing performance parameters can be corrected; (5) Data version management: Version control of performance parameters under different times or different measurement strategies.

[0045] S2 Structured Training Performance Simulation Based on Multidimensional Feature-Based Performance Database After constructing the multidimensional feature-based performance database, this step performs structured performance simulation of the training process based on the training model structure and training configuration, such as... Figure 3 As shown.

[0046] S2.1 Building a Module Hierarchy First, the neural network model structure is hierarchically decomposed to construct a module-level structure. Specifically, this module-level structure is a top-down hierarchical structure, including: (1) Root node, representing the overall model; (2) Intermediate nodes represent functional modules or sub-modules; (3) Leaf nodes represent independent sub-modules that are no longer decomposed.

[0047] In one implementation, a leaf node may contain one or more computational or communication operators. For example, for a Transformer-based model, its module tree structure may include: (1) Transformer root node; (2) Intermediate nodes of Embedding and Transformer Block; (3) Each Transformer Block intermediate node contains several Transformer Layer intermediate nodes; (4) Each intermediate node of the Transformer Layer contains intermediate nodes such as Attention and MLP; (5) Attention, MLP and other intermediate nodes further contain leaf nodes such as linear layers and normalization layers; (6) Each leaf node corresponds to one or more computation operators or communication operators.

[0048] This step involves constructing a module hierarchy, enabling performance prediction results to be reflected not only in the overall dimension but also at the module-level granularity at different levels for statistical analysis and error decomposition.

[0049] S2.2 Extracting Operator Execution Feature Combinations After constructing the module hierarchy, in order to query the corresponding performance parameters in the performance database, it is necessary to obtain the operator execution feature combination in each leaf node.

[0050] The operator execution feature combination is obtained through an automatic parsing mechanism. Specifically, during the simulation phase, the system first reads the model structure definition and training configuration file to obtain the model hierarchical structure, operator parameters, and parallel strategy information. Then, combined with the hardware deployment configuration, it determines the execution environment of each operator on different devices, obtaining the number of devices, device types, and communication topology information. By parsing and mapping the training configuration and hardware deployment configuration, the execution feature combination of each operator can be obtained, and performance parameters can be queried in the performance database accordingly.

[0051] In one implementation, the automatic parsing mechanism includes the following steps: (1) Parse the model structure definition file to obtain the module hierarchy, operator type and operator parameters; (2) Analyze the training configuration parameters to determine the parallel strategy, the method of dividing the computing tasks, and the communication group information; (3) Analyze the hardware deployment configuration to obtain the number of devices, device type, peak computing power of devices, communication topology, communication bandwidth and latency parameters; (4) Generate the operator execution feature combination for each operator based on the above information.

[0052] Through the above-mentioned automatic parsing mechanism, the combination of operator execution features consistent with the actual training environment can be automatically generated without the need for manual specification of operator features. This combination can then be used as an index condition for querying the performance database, supporting automatic querying of performance parameters and performance prediction.

[0053] S2.3 Query the performance database and estimate the sub-execution time. For each operator, based on the operator execution feature combination obtained in step S2.2, matching performance parameters are searched in the performance database constructed in S1. In one implementation: (1) If there is a perfectly matching feature combination, then directly read the corresponding operator execution efficiency parameter; (2) If there is no perfect match, feature approximation matching or interpolation strategies can be used to obtain the estimated operator execution efficiency parameters.

[0054] Subsequently, based on the theoretical computational or communication load of the operator, and combined with the operator execution efficiency parameters obtained from the query, the execution time of the operator is calculated: (1) For computation operators, the execution time of the operator can be calculated based on the theoretical computational load (floating-point operation load), the operator execution efficiency parameters, and the peak computing power of the device; (2) For communication operators, the execution time of the operator can be estimated based on the theoretical communication volume, the operator execution efficiency parameters, the theoretical bandwidth and delay parameters.

[0055] S2.4 Scheduling Modeling and Module-Level Time Aggregation After obtaining the operator execution time of each operator, a scheduling model for the training execution process is constructed based on the module hierarchy and operator execution dependencies, thereby achieving module-level aggregation of operator execution time.

[0056] Specifically, the scheduling model includes two levels: operator-level scheduling modeling and module-level scheduling modeling.

[0057] (I) Operator-level scheduling modeling For each leaf node, the system schedules and models the execution process of operators based on the operator execution dependencies between its internal operators, in order to aggregate the execution time of that leaf module.

[0058] In one implementation, the operator execution dependency includes, but is not limited to, the following cases: (1) First order execution relationship. When multiple operators run sequentially according to the program execution order, the execution time is the sum of the execution times of each operator.

[0059] (2) First parallel execution relationship. When multiple operators are executed in parallel on different threads, their execution time is the maximum value of the execution time of each operator.

[0060] (3) Overlapping relationship between computation and communication. In the distributed training process, some communication operators can be executed simultaneously with computation operators. When there is an overlapping execution relationship, the module execution time is calculated by combining the computation time and communication time according to the degree of overlap.

[0061] (II) Module-level scheduling modeling After obtaining the execution time of each leaf node, the system further models the execution relationship between each node based on the overall hierarchical structure of the model, thereby constructing a complete training execution scheduling model.

[0062] In one implementation, the execution relationships between modules are determined by the model structure and training process, including but not limited to: (1) Second sequential execution relationship: Different nodes are executed sequentially according to the model structure, and the total execution time is the sum of the execution times of each node; (2) Second parallel execution relationship: Different nodes execute in parallel on different devices or computing resources, and the total execution time is the maximum value of the execution time of each node; (3) Pipeline execution relationship: In the pipeline parallel training scenario, different modules are distributed in different stages and overlapped execution is achieved through micro-batch scheduling.

[0063] After completing operator-level and module-level scheduling modeling, the system performs bottom-up aggregation calculations of execution time based on the module hierarchy: (1) Leaf nodes: Execution time is determined by operator-level scheduling results; (2) Intermediate node: The execution time is determined by the execution time of its child nodes according to the module-level scheduling relationship; (3) Root node: The execution time corresponds to the prediction result of the execution time of a single iteration of the overall training.

[0064] Through the above-described hierarchical scheduling modeling and module-level time aggregation process, we can characterize the parallel execution and scheduling behavior during the training process while ensuring modeling accuracy, thereby obtaining module-level performance prediction results that are more consistent with actual execution.

[0065] S3 acquires real training performance data at the module level and performs module-level error analysis. After performance simulation prediction is completed, a portion of representative training configurations can be selected as sampling samples for real training, and the corresponding training performance data obtained are the module-level real training performance data. The module-level real training performance data refers to the module execution time data corresponding to the model module hierarchy structure, collected and organized during the real training process. It includes at least the module name, module hierarchy path, module execution stage (forward propagation or backward propagation), module execution time (distinguishing between computation time and communication time), and execution time percentage. Then, the module-level real training performance data is compared with the module-level performance prediction results to perform module-level error analysis, identifying the sources of performance prediction deviations and providing a basis for subsequent performance calibration.

[0066] S3.1 Obtain module-level real training performance data like Figure 4 As shown, this step uses a training performance analysis tool to collect performance data from the actual training process, obtaining statistical information on the execution time of each module of the model. The training performance analysis tool can identify module calls during the training process based on runtime event log data, performance tracking data, or system monitoring information, and collect the execution time of each module.

[0067] By performing module-level performance analysis on the training execution process, the system first obtains the performance acquisition results during the actual training process. These performance acquisition results include at least module call events, computation kernel events, communication kernel events, event start times, event durations, and correlation information between events.

[0068] Subsequently, the system organizes the performance acquisition results according to the predetermined model hierarchy. Specifically, the system identifies the execution interval corresponding to each module during the actual training process based on the module name, module hierarchy path, and event time range of the module call event; then, it assigns the computation events, communication events, or other runtime events located within the execution interval to the corresponding module, and calculates the computation time, communication time, and total execution time of each module.

[0069] When the model hierarchy includes a root node, multiple intermediate child nodes, and leaf nodes, the system can first calculate the time cost of the leaf nodes, and then summarize it from bottom to top to the intermediate child nodes and the root node according to the parent-child hierarchy relationship, thereby obtaining the time cost distribution of different level modules in the actual training process.

[0070] After the above processing, module-level real training performance data is generated. This module-level real training performance data includes at least the module name, module hierarchical path, module execution stage (forward propagation or backward propagation), module execution time (distinguishing between computation time and communication time), and the percentage of execution time. This module-level real training performance data is used to compare with the module-level performance prediction results obtained in the performance simulation stage to support subsequent module-level error analysis.

[0071] S3.2 Execution Module-Level Error Analysis After obtaining the module-level real training performance data, the system compares the module-level real training performance data with the module-level performance prediction results obtained in step S2, and performs module-level error analysis. The module-level error analysis is used to identify the differences between the performance simulation prediction results and the actual training results, and to determine the model modules with larger errors and their possible sources.

[0072] First, the system establishes a module-level correspondence between real training performance data and module-level performance prediction results. This module correspondence refers to the matching relationship between the real modules collected during real training and the predicted modules generated during performance simulation. Since both the real training and performance simulation phases are based on the same or corresponding neural network model structures, the correspondence between them can be established according to the model module hierarchy.

[0073] In one implementation, the module correspondence is established in the following way: (1) The module hierarchy structure generated in the performance simulation stage and the module hierarchy structure obtained in the real training stage are standardized to obtain a module hierarchy path in a unified format. (2) Match the predicted modules and the real modules according to the module name and module hierarchical path; (3) When the module names are completely identical and the module hierarchical paths are identical, establish an exact module correspondence; (4) When there are differences in frame prefix, numbering or naming format in the module name, the module name shall be standardized before the corresponding relationship is established; (5) When a prediction module corresponds to multiple real sub-modules, or multiple prediction modules correspond to one real module, an aggregate correspondence relationship is established based on the module hierarchical path and the parent-child node relationship.

[0074] By establishing the above module correspondence, the simulation prediction results and the actual training results can be aligned to the same model semantic level, providing a basis for module-level error analysis.

[0075] After establishing the module correspondence, error analysis is performed on the module-level actual training performance data and module-level performance prediction results. In one implementation, at least one of the following errors can be calculated for each module: (1) Absolute error, which is the absolute value of the difference between the module execution time collected in the actual training and the module execution time predicted in the simulation; (2) Relative error, which is the ratio of absolute error to the actual module execution time collected during training; (3) Error contribution, which is used to characterize the degree of influence of the module error on the overall training time error.

[0076] After obtaining the error indices for each module, the errors are summarized layer by layer according to the model's hierarchical structure to obtain the error hierarchy distribution result. Specifically, the error situation of the leaf nodes can be analyzed first, and then summarized upwards to the intermediate nodes and the root node, thus obtaining the error distribution of modules at different levels, which is the error hierarchy distribution result. Through this process, key modules with large errors and their hierarchical positions can be identified.

[0077] Furthermore, to analyze the sources of error, the error hierarchy distribution results can be classified and attributed by combining the composition of operators within the module, the module execution stage, and the corresponding computational or communication characteristics of the module to obtain error attribution results. In one embodiment, the error attribution results include, but are not limited to: (1) Calculate the deviation of operator performance parameters; (2) Deviation in the performance parameters of the communication operator; (3) The modeling of module execution scheduling relationships is inaccurate; (4) No operators were modeled, and execution behavior was not covered; (5) Insufficient feature space modeling.

[0078] Finally, the system generates module-level error analysis results based on the errors of each module, the error hierarchy distribution results, and the error attribution results. These module-level error analysis results may include: error module identifier, module hierarchy position, module prediction time, module actual time, error value, error proportion, and error source category, thus providing a basis for subsequent modeling correction and performance calibration.

[0079] S4 performs calibration optimization based on error analysis. like Figure 5 As shown, after obtaining the module-level error analysis results in step S3, this step aims to perform model calibration and optimization on the performance simulation model or performance database, thereby reducing performance prediction errors and improving the prediction accuracy of the simulation model.

[0080] Specifically, the model calibration optimization includes adjusting the benchmark measurement method and correcting the performance simulation model structure.

[0081] S4.1 Adjustment of reference measurement method When module-level error analysis results indicate significant deviations in the performance modeling of certain operators, the system will adjust the benchmark measurement method for the corresponding operator. Specifically, the adjustment of the benchmark measurement method includes the following methods: (1) Expanding the test parameter range of operators: During the initial benchmark measurement process, operator testing can usually only cover a preset limited range of input parameters. When the operator input size in the real training task exceeds the range, the operator execution efficiency parameters in the performance database may not accurately reflect the actual execution performance. At this time, the test parameter range of operators is expanded according to the module-level error analysis results, and the benchmark measurement is re-executed to obtain operator execution efficiency that is closer to the real training scenario. For example, for matrix multiplication operators, the test parameter set can be expanded according to the matrix size that appears in the actual training.

[0082] (2) Increase the sampling density of feature regions: In the initial benchmark measurement stage, in order to control measurement costs, only a limited number of sampling points are usually selected in the feature space for testing. When the number of sampling points is insufficient to capture the fluctuations in performance with feature changes, the prediction accuracy of the performance model in some feature regions may be low. Therefore, new sampling points can be added in feature regions with large errors and the operator benchmark test can be re-executed to improve the accuracy of the performance database in characterizing the performance features of that region.

[0083] (3) Increase the feature dimension of the operator: In some cases, the performance parameters of the operator are not only affected by the input size, but also related to other execution features. For example, for the matrix multiplication operator, its execution efficiency is not only related to the shape of the input tensor, but may also be affected by factors such as the memory layout of the input tensor. Therefore, new feature dimensions can be introduced in the process of operator performance modeling, so that the operator performance database can store the execution efficiency parameters in a higher-dimensional feature space, thereby improving the accuracy of performance modeling.

[0084] S4.2 Simulation Model Structure Correction When module-level error analysis results indicate that the performance prediction error of some modules originates from an incomplete simulation model structure, the simulation model can be structurally corrected to improve the accuracy of training performance simulation.

[0085] Performance simulation models typically only model the key computational or communication operators within a module, while omitting simulation models for operators with shorter execution times or those considered to have a minor impact on overall performance. However, in actual training, these unmodeled operators can lead to significant errors between the actual trained module execution time and the simulation-predicted module execution time in certain scenarios.

[0086] Therefore, based on the module-level error analysis results, the simulation model is structurally modified: (1) Performance model for supplementing missing operators In one implementation, when the error attribution result indicates that the error of a certain module originates from an unmodeled operator, it can be identified as an important operator in that module that is not included in the simulation model, and the unmodeled operator can be added to the simulation model.

[0087] For example, in some neural network modules, the original simulation model may only model the main computational operators such as matrix multiplication, while ignoring auxiliary operators such as normalization operators, activation functions, or tensor transformations. During error analysis, if it is found that these operators have a non-negligible impact on the module's execution time, they can be added to the module's simulation model, and a corresponding performance parameter model can be established for that operator.

[0088] (2) Modify the operator scheduling model of the simulation model The execution time of a module in real training depends not only on the composition of the operators within the module, but also on the execution scheduling relationship between the operators. In the initial performance simulation model, the scheduling modeling of operators within the module is usually based on preset rules, such as the overlap between computation and communication, the parallel execution relationship between operators, and the pipelined execution structure.

[0089] However, during actual training, the actual execution relationships of operators within a module may differ from the assumptions in the simulation model under different model structures, parallel strategies, or hardware environments, leading to module-level performance prediction errors. Therefore, the operator-level scheduling model in the initial performance simulation model can be structurally adjusted based on the error attribution results.

[0090] In one implementation, the scheduling structure adjustment may include the following methods: (1) Adjust the overlap relationship between the computation operator and the communication operator; (2) Adjust the parallel execution relationship between operators within the module; (3) Adjusting the stage scheduling relationship in the pipeline parallel scenario.

[0091] By using the above methods, the module execution scheduling structure in the simulation model can be made more consistent with the actual training execution behavior, thereby improving the prediction accuracy of training performance simulation.

[0092] S4.3 Performance Model Iterative Optimization After adjusting the benchmark measurement method and correcting the simulation model structure, the performance database or the initial performance simulation model structure is updated, and the training performance simulation is re-executed to enter the iterative optimization phase.

[0093] Specifically, the system can re-predict the performance of the target training task based on the adjusted benchmark measurement method or the revised simulation model structure, and obtain new module-level performance simulation results. Subsequently, a portion of the training configuration can be selected again for real training runs, and the corresponding module-level performance data can be obtained to evaluate the prediction accuracy of the updated simulation model.

[0094] If the new error analysis results still indicate that some modules have large prediction biases, the performance database or simulation model can be further adjusted, and the performance simulation and error analysis process can be repeated.

[0095] Through the above-described measurement-simulation-prediction-correction approach, this invention establishes an iterative optimization mechanism between benchmark measurement, performance simulation, and error analysis, enabling the performance simulation model to gradually improve its prediction accuracy under different model structures, training scales, and hardware environments.

[0096] Example 2: A performance simulation and closed-loop calibration system Secondly, the present invention also provides a performance simulation and closed-loop calibration system for implementing the above-mentioned method, such as... Figure 6 As shown, it includes a performance feature construction unit, a performance simulation prediction unit, a module-level error analysis unit, and a model calibration and optimization unit.

[0097] The performance feature construction unit is used to perform multi-dimensional feature modeling of the execution behavior of computational and communication operators. In the computational dimension, in addition to operator type, features such as input tensor shape, data layout, and parallel partitioning method can be introduced; in the communication dimension, features such as communication data volume, communication group size, intra-machine or inter-machine topology, bandwidth, and latency parameters can be introduced. Operator execution efficiency parameters are measured through the multi-dimensional feature space, forming a structured and storable set of performance parameters.

[0098] The performance simulation prediction unit performs structured simulation modeling of the neural network training process based on the performance parameters in the multidimensional characteristic performance database, and outputs module-level performance prediction results and overall training time prediction results.

[0099] The module-level error analysis unit is used to acquire module-level real training performance data, compare the module-level real training performance data with the module-level performance prediction results, and generate module-level error analysis results.

[0100] The model calibration and optimization unit executes an error-driven modeling correction strategy based on the error analysis results, including but not limited to expanding the performance feature dimensions, adjusting the benchmark measurement strategy, supplementing unmodeled operators, or correcting simulation structure assumptions, thereby achieving structural optimization of the simulation model.

[0101] like Figure 7As shown, this application provides an electronic device including a memory 101 for storing one or more programs and a processor 102. When the one or more programs are executed by the processor 102, they implement the method as described in any of the first aspects above.

[0102] The system also includes a communication interface 103. The memory 101, processor 102, and communication interface 103 are electrically connected directly or indirectly to each other to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used for signaling or data communication with other node devices.

[0103] The memory 101 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0104] The processor 102 can be an integrated circuit chip with signal processing capabilities. The processor 102 can be a general-purpose processor 102, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0105] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can also be implemented in other ways. The method and system embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0106] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0107] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program thereon. When executed by processor 102, the computer program implements the methods described in any of the first aspects above. If the functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

Claims

1. A performance simulation and closed-loop calibration method for distributed neural network training, characterized in that, Includes the following steps: S1: Construct a multi-dimensional performance feature space for computational and communication operators; generate operator execution feature combinations based on the multi-dimensional performance feature space; then perform benchmark measurements to construct operator execution efficiency parameters; store the operator execution feature combinations and operator execution efficiency parameters in a structured manner to establish a multi-dimensional characteristic performance database; the multi-dimensional performance feature space is specifically a feature set used to describe the operating characteristics of operators under different execution conditions. The multi-dimensional performance feature space is predefined according to the operator type, training configuration parameters, parallel strategy, hardware platform information, and communication environment information in the neural network model, or is automatically generated after parsing the model structure and training configuration; The multidimensional performance feature space includes a computation operator feature space and a communication operator feature space. The computation operator feature space includes, but is not limited to, operator type, input tensor shape, data memory layout, and operating hardware platform information. The communication operator feature space includes, but is not limited to, communication operator type, communication data volume, communication participant group size, and communication bandwidth and latency parameters. The operator execution efficiency parameter characterizes the actual execution efficiency of the operator under a specific combination of operator execution features. For computation operators, the operator execution efficiency parameter represents the ratio between actual computational performance and the theoretical peak computational capability of the hardware. For communication operators, the operator execution efficiency parameter represents the ratio between the effective bandwidth and the theoretical bandwidth during communication. S2: Construct a module hierarchy based on the neural network model structure, extract operator execution feature combinations based on model structure information, training configuration parameters, and hardware deployment configuration; query the corresponding performance parameters in the multidimensional feature-based performance database, and predict the operator execution time based on the queried performance parameters; aggregate the operator execution time based on the module hierarchy and scheduling model to obtain module-level performance prediction results; S3: Obtain module-level real training performance data during the actual training process; establish the module correspondence between module-level real training performance data and module-level performance prediction results, compare the module-level real training performance data with the module-level performance prediction results, perform module-level error analysis, and obtain module-level error analysis results; S4: Based on the module-level error analysis results, the benchmark measurement method or performance simulation model is calibrated and optimized, and the structured training performance simulation is re-executed, thereby forming an iterative optimization closed loop between performance measurement, performance simulation, error analysis and calibration.

2. The method according to claim 1, characterized in that, In step S1, The multidimensional characteristic performance database is obtained by structurally storing the operator execution efficiency parameters according to the operator execution feature combination; the multidimensional characteristic performance database is used to establish the mapping relationship between the operator execution feature combination and the operator execution efficiency parameters. The multidimensional characteristic performance database includes multiple data records, each data record corresponding to a set of operator execution feature combinations and their corresponding performance parameters.

3. The method according to claim 1, characterized in that, In step S2, the operator execution feature combination is obtained through an automatic parsing mechanism. The performance parameters are a set of data that characterizes the operator performance under the corresponding operator execution feature combination. These parameters include operator execution efficiency parameters, statistical results of raw measurement data, stable performance measurement values, theoretical computational load, theoretical communication load, measurement hardware environment identifier, and version number of the multidimensional characteristic performance database. The automatic parsing mechanism includes: parsing the model structure definition file to obtain the module hierarchy, operator type, and operator parameters; parsing the training configuration parameters to obtain the parallel strategy, computation task partitioning method, and communication group information; parsing the hardware deployment configuration to obtain the device type, number of devices, peak computing power of devices, communication topology, communication bandwidth, and latency parameters; and generating operator execution feature combinations for each operator based on the above parsing results during structured training performance simulation, and querying the corresponding operator execution efficiency parameters in the multidimensional feature-based performance database according to the operator execution feature combinations. The specific execution time of the operator is as follows: for computation operators, the execution time is calculated based on the theoretical computational load, operator execution efficiency parameters, and the peak computing capacity of the device; for communication operators, the execution time is calculated based on the theoretical communication load, operator execution efficiency parameters, theoretical bandwidth, and latency parameters.

4. The method according to claim 1, characterized in that, In step S2, The aggregation of operator execution time based on module hierarchy and scheduling model specifically involves: during module-level time aggregation, a scheduling model is established based on the module hierarchy and operator execution dependencies; then, the execution time is aggregated from bottom to top based on the module hierarchy to obtain the module-level performance prediction result; the scheduling model includes operator-level scheduling modeling and module-level scheduling modeling. In the operator-level scheduling modeling, the operator execution dependency includes at least one of the following: a first sequential execution relationship; a first parallel execution relationship; and a computation and communication overlapping execution relationship. In the module-level scheduling modeling, the module execution dependency relationship includes at least one of the following: a second sequential execution relationship; a second parallel execution relationship; and a pipelined parallel execution relationship. When the execution relationship is first or second sequential, the execution time is the sum of the execution times of the corresponding subtasks. When the execution relationship is first or second parallel, the execution time is the maximum value of the execution times of the parallel subtasks, or the sum of the maximum value and the synchronization overhead. When the execution relationship is computation and communication overlapping, the execution time is determined based on the computation time, communication time, and overlap ratio. When the execution relationship is pipelined, the overall execution time is determined based on the number of pipeline stages, the number of micro-batches, the execution time of each stage, and the scheduling strategy.

5. The method according to claim 1, characterized in that, In step S3, the module-level real training performance data specifically refers to the module execution time data corresponding to the model module hierarchical structure, which is collected and organized during the real training process. The module-level error analysis establishes a module correspondence between module-level performance prediction results and module-level real training performance data based on the hierarchical structure of the neural network model modules. Error analysis is performed on the module-level real training performance data and module-level performance prediction results to obtain the errors of each module. The errors are then summarized layer by layer according to the model hierarchy to obtain the error hierarchy distribution results. Furthermore, the error hierarchy distribution results are classified and attributed based on the module's internal operator composition, module execution stage, and corresponding computational or communication characteristics to obtain error attribution results. Finally, based on the errors of each module, the error hierarchy distribution results, and the error attribution results, the module-level error analysis results are generated. Specifically, the error hierarchy distribution result is obtained by first analyzing the error situation of the leaf nodes, and then summarizing it up to the intermediate nodes and the root node, thereby obtaining the error distribution of different level modules, i.e., the error hierarchy distribution result. The module correspondence is established in the following way: (1) The module hierarchy structure generated in the performance simulation stage and the module hierarchy structure obtained in the real training stage are standardized to obtain a module hierarchy path in a unified format. (2) Match the predicted modules and the real modules according to the module name and module hierarchical path; (3) When the module names are completely identical and the module hierarchical paths are identical, establish an exact module correspondence; (4) When there are differences in frame prefix, numbering or naming format in the module name, the module name shall be standardized before the corresponding relationship is established; (5) When a prediction module corresponds to multiple real sub-modules, or multiple prediction modules correspond to one real module, an aggregate correspondence relationship is established based on the module hierarchical path and the parent-child node relationship; The error includes: (1) absolute error, which is the absolute value of the difference between the module execution time collected in the actual training and the module execution time predicted in the simulation; (2) Relative error, which is the ratio of absolute error to the actual module execution time collected during training; (3) Error contribution, used to characterize the degree of influence of module error on the overall training time error; The error attribution results include: (1) Calculate the deviation of operator performance parameters; (2) Deviation in the performance parameters of the communication operator; (3) The modeling of module execution scheduling relationships is inaccurate; (4) No operators were modeled, and execution behavior was not covered; (5) Insufficient feature space modeling.

6. The method according to claim 1, characterized in that, In step S4, model calibration optimization includes adjusting the benchmark measurement method and correcting the performance simulation model structure; wherein, the benchmark measurement method adjustment includes, but is not limited to: expanding the test parameter range of operators; increasing the sampling density of feature regions; and increasing the feature dimension of operators; the performance simulation model structure correction includes correcting the performance simulation model structure, and the correction includes, but is not limited to: supplementing the performance model of missing operators; and correcting the operator scheduling modeling of the performance simulation model.

7. A performance simulation and closed-loop calibration system for distributed neural network training based on the method described in any one of claims 1 to 6, characterized in that, include: The performance feature construction unit is used to construct a multi-dimensional performance feature space for computational and communication operators, and to obtain operator execution efficiency parameters through benchmark measurements to form a multi-dimensional characteristic performance database. The performance simulation prediction unit is used to perform performance simulation prediction based on the neural network model structure and training configuration, and output module-level performance prediction results. The module-level error analysis unit is used to acquire module-level real training performance data during the actual training process, compare it with the module-level performance prediction results, perform module-level error analysis, and obtain module-level error analysis results. The model calibration and optimization unit is used to adjust the performance database or performance simulation model based on the module-level error analysis results, and re-execute the performance simulation prediction to achieve closed-loop optimization of the performance simulation model.

8. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the performance simulation and closed-loop calibration method for distributed neural network training as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the performance simulation and closed-loop calibration method for distributed neural network training as described in any one of claims 1-6.

Images