A method for determining runtime performance of a deep learning training task and related products

Abstract

This application discloses a method and related products for determining the runtime performance of deep learning training tasks. The scheme involves acquiring the original runtime events of the task to be processed and constructing a fine-grained computation graph based on the execution trajectory sequence corresponding to the original runtime events; extracting multi-dimensional features from the fine-grained computation graph to obtain an initial feature matrix; employing a nested cross-validation mechanism to match a corresponding predictor for each performance indicator in a preset performance indicator set; the nested cross-validation mechanism includes outer cross-validation and inner cross-validation; outer cross-validation is used to determine the prediction model corresponding to each performance indicator; inner cross-validation is used to determine the feature subset and hyperparameter combination corresponding to each performance indicator; and runtime performance prediction of the task to be processed is performed based on the predictors that correspond one-to-one with multiple performance indicators in the preset performance indicator set. Compared to existing technologies where the performance prediction results deviate significantly from the actual running conditions, this application has significant advantages.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method for determining the runtime performance of a deep learning training task and related products. Background Technology

[0002] Deep learning training is an iterative optimization process based on gradient descent, aiming to dynamically adjust model parameters by minimizing the loss function to achieve a preset training objective. The runtime performance of deep learning training tasks, such as training efficiency, resource consumption, and convergence speed, directly determines the execution cost and feasibility of deep learning training. Existing methods for determining runtime performance in deep learning training tasks largely rely on simplified analysis of model layer structures. The drawback of this approach is that it oversimplifies the complex runtime behavior of deep learning training, treating operators during training as indivisible execution black boxes. This fails to penetrate the abstract encapsulation of the deep learning framework to characterize the internal micro-competitions. Especially in modern deep learning training scenarios, compilers often employ aggressive optimization strategies such as operator fusion and memory reorganization, further exacerbating the complexity of runtime behavior. Because existing methods cannot perceive the dynamic changes brought about by compiler optimizations, such as instruction stream reorganization and deep reuse of memory access paths, they are prone to falling into the logical illusion that the predicted object no longer exists, leading to significant deviations between performance predictions and actual runtime conditions.

[0003] How to accurately predict the runtime performance of deep learning training tasks is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a method and related products for determining the runtime performance of deep learning training tasks, with the aim of improving the accuracy of runtime performance prediction results for deep learning training tasks.

[0005] The embodiments of this application disclose the following technical solutions: The first aspect of this application provides a method for determining the runtime performance of a deep learning training task, the method comprising: Obtain the original runtime events of the task to be processed, and construct a fine-grained computation graph based on the execution trajectory sequence corresponding to the original runtime events; Extract the multi-dimensional features of the fine-grained computation graph to obtain an initial feature matrix; the multi-dimensional features include node attribute features, edge attribute features, environment perception features, hyperparameter features, and graph topology features; A nested cross-validation mechanism is employed to match a corresponding predictor for each performance indicator in a preset set of performance indicators. The predictor includes a prediction model, a feature subset, and a hyperparameter combination. The nested cross-validation mechanism includes an outer cross-validation layer and an inner cross-validation layer. The outer cross-validation layer is used to determine the prediction model corresponding to each performance indicator. The inner cross-validation layer is used to determine the feature subset and hyperparameter combination corresponding to each performance indicator. The runtime performance of the task to be processed is predicted based on a predictor that corresponds one-to-one with multiple performance indicators in the preset performance indicator set, and the performance prediction result is obtained.

[0006] Optionally, the nested cross-validation mechanism, which matches a corresponding predictor to each performance indicator in a preset set of performance indicators, includes: The preset model table is traversed, and the feature subsets and hyperparameter combinations corresponding to each performance index obtained by the inner cross-validation are used to train and optimize multiple candidate models in the preset model table on the training set, and the performance is evaluated on the test set. The prediction model corresponding to each performance index is determined based on the performance evaluation results.

[0007] Optionally, the nested cross-validation mechanism, which matches a corresponding predictor to each performance indicator in a preset set of performance indicators, includes: Using multiple dimensionality reduction strategies, the initial feature matrix is ​​dimensionality reduced according to multiple performance indicators in a preset set of performance indicators, to obtain the feature subset of each performance indicator under each dimensionality reduction strategy; Based on preset screening criteria, the feature subsets of each performance indicator under multiple dimensionality reduction strategies are screened to determine the feature subset corresponding to each performance indicator; Hyperparameter optimization is performed based on the feature subset corresponding to each performance index to obtain the hyperparameter combination corresponding to each performance index.

[0008] Optionally, the multiple dimensionality reduction strategies include a three-level progressive dimensionality reduction strategy. The step of using multiple dimensionality reduction strategies to perform dimensionality reduction processing on the initial feature matrix according to multiple performance indicators in a preset performance indicator set, obtaining a feature subset of each performance indicator under each dimensionality reduction strategy, includes: Using a three-level progressive dimensionality reduction strategy, the initial feature matrix is ​​dimensionality reduced according to multiple performance indicators in a preset performance indicator set, to obtain a feature subset of each performance indicator under the three-level progressive dimensionality reduction strategy; the three-level progressive dimensionality reduction strategy includes feature selection, feature compression and collinearity elimination in sequence.

[0009] Optionally, the three-level progressive dimensionality reduction strategy is used to reduce the dimensionality of the initial feature matrix according to multiple performance indicators in a preset performance indicator set, to obtain a feature subset of each performance indicator under the three-level progressive dimensionality reduction strategy, including: The initial feature matrix is ​​saliency-based by random forest feature importance to obtain the first-level dimensionality reduction result; The first-level dimensionality reduction result is sparsely compressed by L1 regularization, and then optimized according to the preset objective function to obtain the second-level dimensionality reduction result. The multicollinearity among the features of the second-level dimensionality reduction result is quantified by the variance inflation factor, and features exceeding a preset threshold are removed based on the quantification result to obtain the third-level dimensionality reduction result.

[0010] Optionally, the multiple dimensionality reduction strategies include a recursive feature elimination strategy. The step of using multiple dimensionality reduction strategies to perform dimensionality reduction processing on the initial feature matrix according to multiple performance indicators in a preset performance indicator set, obtaining a feature subset for each performance indicator under each dimensionality reduction strategy, includes: Using a recursive feature elimination strategy, the initial feature matrix is ​​dimensionality-reduced based on multiple performance indicators in a preset performance indicator set to obtain a feature subset for each performance indicator under the recursive feature elimination strategy. The recursive feature elimination strategy is used to perform multiple iterations of filtering on the initial feature matrix based on feature importance scores. In each iteration, the feature with the lowest current contribution is removed. Through multi-fold cross-validation, negative mean square error is used as the evaluation index to obtain the feature subset with the best generalization performance.

[0011] Optionally, the multi-dimensional features include graph topological features, and the extraction of multi-dimensional features from the fine-grained computation graph to obtain the initial feature matrix includes: Based on the scale of the fine-grained computation graph, a corresponding graph topology feature extraction strategy is determined; the graph topology feature extraction strategy includes feature analysis depth and sampling ratio. The graph topology features of the fine-grained computation graph are extracted using the graph topology feature extraction strategy to obtain the graph topology feature extraction results.

[0012] Optionally, constructing a fine-grained computation graph based on the execution trajectory sequence corresponding to the original runtime events includes: The original runtime events are filtered using preset rules to obtain filtered runtime events; The filtered runtime events are constructed based on the call stack and timestamp information to obtain a fine-grained computation graph.

[0013] A second aspect of this application provides an apparatus for determining runtime performance of a deep learning training task, the apparatus comprising: The graph construction module is used to obtain the original runtime events of the task to be processed, and to construct a fine-grained computation graph based on the execution trajectory sequence corresponding to the original runtime events. A multi-dimensional feature extraction module is used to extract multi-dimensional features of the fine-grained computation graph to obtain an initial feature matrix; the multi-dimensional features include node attribute features, edge attribute features, environment perception features, hyperparameter features, and graph topology features; A predictor determination module is used to match a corresponding predictor for each performance indicator in a preset set of performance indicators using a nested cross-validation mechanism. The predictor includes a prediction model, a feature subset, and a hyperparameter combination. The nested cross-validation mechanism includes outer cross-validation and inner cross-validation. The outer cross-validation is used to determine the prediction model corresponding to each performance indicator. The inner cross-validation is used to determine the feature subset and hyperparameter combination corresponding to each performance indicator. The performance prediction module is used to perform runtime performance prediction on the task to be processed based on a predictor that corresponds one-to-one with multiple performance indicators in the preset performance indicator set, and obtain the performance prediction result.

[0014] A third aspect of this application provides a processor for running a computer program that, when running, executes a method for determining runtime performance of a deep learning training task as provided in any implementation of the first aspect.

[0015] Compared with the prior art, this application has the following beneficial effects: The deep learning training task runtime performance determination method provided in this application embodiment obtains the original runtime events of the task to be processed, and constructs a fine-grained computation graph based on the execution trajectory sequence corresponding to the original runtime events; extracts multi-dimensional features of the fine-grained computation graph to obtain an initial feature matrix; the multi-dimensional features include node attribute features, edge attribute features, environment perception features, hyperparameter features, and graph topology features; a nested cross-validation mechanism is used to match a corresponding predictor for each performance indicator in a preset performance indicator set; the predictor includes a prediction model, a feature subset, and a hyperparameter combination; the nested cross-validation mechanism includes outer cross-validation and inner cross-validation; the outer cross-validation is used to determine the prediction model corresponding to each performance indicator; the inner cross-validation is used to determine the feature subset and hyperparameter combination corresponding to each performance indicator; and the runtime performance of the task to be processed is predicted based on the predictors that correspond one-to-one with the multiple performance indicators in the preset performance indicator set to obtain the performance prediction result.

[0016] The above method constructs a fine-grained computation graph based on the execution trajectory sequence of the original runtime events. This fully preserves the underlying runtime details, such as operator execution order, data dependencies, memory read / write, and communication patterns, avoiding prediction bias caused by information loss and laying a high-quality feature foundation for accurate runtime performance prediction. Extracting multi-dimensional features from the fine-grained computation graph allows for a more comprehensive capture of the impact of various factors on runtime performance, significantly reducing prediction inaccuracies caused by incomplete features and improving the reliability and accuracy of runtime performance prediction results. A nested cross-validation mechanism is employed, using outer cross-validation to optimize the prediction model and inner cross-validation to select the optimal feature subset and hyperparameter combination. This achieves fine-grained adaptation between each performance metric and the predictor, significantly improving the accuracy of the overall performance prediction results. Joint prediction based on dedicated predictors corresponding one-to-one with multiple performance metrics accurately captures the changing trends of different performance metrics, avoiding mutual interference between them. Even in complex deep learning training scenarios, it maintains high prediction accuracy, making the final performance prediction results closer to real runtime performance. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a method for determining runtime performance of a deep learning training task, provided in an embodiment of this application; Figure 2 A flowchart of a fine-grained computational graph scale adaptive analysis provided in this application embodiment; Figure 3 A flowchart illustrating a nested cross-validation method provided in this application embodiment; Figure 4 This is a schematic diagram of a device for determining the runtime performance of a deep learning training task, provided in an embodiment of this application. Detailed Implementation

[0019] As described earlier, current methods for determining runtime performance in deep learning training tasks largely rely on simplified analysis of the model layer structure. The drawback of this approach is that it oversimplifies the complex runtime behavior of deep learning training, treating operators during training as indivisible execution black boxes. This fails to penetrate the abstract encapsulation of the deep learning framework to characterize the internal micro-competitions. Especially in modern deep learning training scenarios, compilers often employ aggressive optimization strategies such as operator fusion and memory reorganization, further exacerbating the complexity of runtime behavior. Because existing methods cannot perceive the dynamic changes brought about by compiler optimizations, such as instruction stream reorganization and deep reuse of memory access paths, they are prone to falling into the logical illusion that the predicted object no longer exists, leading to significant deviations between performance predictions and actual runtime conditions.

[0020] In view of the above problems, this application proposes a method and related products for determining the runtime performance of deep learning training tasks. The method involves obtaining the original runtime events of the task to be processed and constructing a fine-grained computation graph based on the execution trajectory sequence corresponding to the original runtime events. Multi-dimensional features of the fine-grained computation graph are extracted to obtain an initial feature matrix. These multi-dimensional features include node attribute features, edge attribute features, environment awareness features, hyperparameter features, and graph topology features. A nested cross-validation mechanism is used to match a corresponding predictor for each performance indicator in a preset performance indicator set. The predictor includes a prediction model, a feature subset, and a hyperparameter combination. The nested cross-validation mechanism includes outer cross-validation and inner cross-validation. The outer cross-validation is used to determine the prediction model corresponding to each performance indicator. The inner cross-validation is used to determine the feature subset and hyperparameter combination corresponding to each performance indicator. Based on the predictors that correspond one-to-one with the multiple performance indicators in the preset performance indicator set, the runtime performance of the task to be processed is predicted to obtain the performance prediction result.

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0022] See Figure 1 This figure is a flowchart of a method for determining the runtime performance of a deep learning training task according to an embodiment of this application. Figure 1 As shown, the method includes the following steps: S101. Obtain the original runtime events of the task to be processed, and construct a fine-grained computation graph based on the execution trajectory sequence corresponding to the original runtime events.

[0023] In one feasible implementation, a fine-grained computation graph is constructed based on the execution trajectory sequence corresponding to the original runtime events, including: The original runtime events are filtered using preset rules to obtain filtered runtime events.

[0024] The filtered runtime events are constructed based on the call stack and timestamp information to obtain a fine-grained computation graph.

[0025] The method for obtaining the raw runtime events of the task to be processed is as follows: perform one iteration on the user-submitted task to be predicted, such as a batch size, use the model tracking function to track each operation, record the input and output tensors of each layer of operation, and generate the full-process operation graph corresponding to the task; call the underlying Profiler interface, which can be configured with with_shapes=True and fully expanded mode, to capture atomic-level event streams in real time, such as GPU kernel startup and memory operations, and obtain the raw execution trajectory sequence containing raw nanosecond-level timestamps, input and output tensor shapes and operator semantics.

[0026] The default rule here is a two-layer filtering rule based on the prefix and backpropagation identifier, which retains only the aten:: computation operator and the autograd back gradient operation, and removes redundant control flow events.

[0027] Fine-grained computation graphs utilize timestamps and call stack information to piece together isolated operator nodes into directed acyclic graphs with semantic and temporal dependencies.

[0028] S102. Extract the multi-dimensional features of the fine-grained computation graph to obtain the initial feature matrix.

[0029] The multi-dimensional features include node attribute features, edge attribute features, environment perception features, hyperparameter features, and graph topology features.

[0030] Node attribute features are key attributes of all low-level operator nodes in the fine-grained computation graph, such as Central Processing Unit (CPU) time, Compute Unified Device Architecture (CUDA) time, execution time, input shape, CPU memory usage, GPU memory usage, floating-point operations, and operator type. Global statistics are performed on these attributes, such as mean, maximum, minimum, standard deviation, quantiles, and frequency of occurrence, transforming local operator characteristics into a global view.

[0031] Edge attribute statistical features are used to count the types of edges that connect the dependencies between nodes, including the number and proportion of each type of edge, to characterize the data transmission pattern.

[0032] Environment-aware features capture the resource status of the host environment before a task runs, such as CPU utilization, GPU utilization, available video memory, and available system memory, reflecting runtime resource constraints.

[0033] Hyperparameter features include key configuration parameters that affect the training process, such as batch size, number of training epochs, learning rate, number of parallel threads for data loading (num_workers), and total number of iterations (iteration).

[0034] Graph topology features are determined by calculating the graph node size with fine granularity to establish the graph topology feature extraction strategy. The graph node size is divided into multiple levels, such as small, medium, and large, and corresponding graph topology feature extraction strategies are preset. For example, a k-random sampling strategy is used for large graphs. Figure 2 As shown, Figure 2 This demonstrates a fine-grained computational graph scale-adaptive analysis workflow. Figure 2 The algorithm categorizes the node size of the input fine-grained computational graph. Small graphs (less than 500 nodes) are analyzed using a full-graph analysis strategy. The core approach is full topological completion analysis with a time complexity of O(n+m), directly performing a complete topological analysis of the entire graph with low and controllable time complexity. Medium graphs (500-2000 nodes) balance global feature extraction with local computational efficiency. This is achieved by sampling 10% of the key subgraphs and setting convergence conditions based on the power-law distribution characteristics of the data, ensuring feature integrity while reducing computational load. Large graphs (2000-2000 nodes) have a maximum of 2000 nodes. For graphs with 00-5000 nodes, a hierarchical mechanism is introduced, sampling 20% ​​of the nodes and combining random walk and other graph algorithms to estimate the steady-state distribution of the critical path, thereby providing an overview of the entire graph structure. For very large graphs with 5000-10000 nodes, a 10% uniform sampling method is used, and the central limit theorem is used to approximate the overall properties of the entire graph through degree distribution. For very large graphs with more than 10000 nodes, 5000 samples are fixedly extracted for lightweight statistics, and the law of large numbers is used to ensure the consistency between the sample statistical results and the overall characteristics of the entire graph, achieving efficient approximate analysis of very large graphs.

[0035] In one feasible implementation, the multi-dimensional features include graph topological features. Extracting the multi-dimensional features of the fine-grained computation graph yields an initial feature matrix, including: Based on the scale of the fine-grained computation graph, a corresponding graph topology feature extraction strategy is determined; the graph topology feature extraction strategy includes feature analysis depth and sampling ratio.

[0036] The graph topology features of the fine-grained computation graph are extracted using the graph topology feature extraction strategy to obtain the graph topology feature extraction results.

[0037] The multi-dimensional features of the fine-grained computation graph are concatenated in high dimension to obtain the initial feature matrix.

[0038] This scale-adaptive graph topology feature extraction strategy dynamically adjusts the feature analysis depth and sampling ratio according to the size of the computation graph nodes to reduce the computational complexity of high-order topology features and ensure low latency in the prediction process.

[0039] By fusing multi-dimensional features at the feature level in the same high-dimensional space, performance fluctuations caused by resource competition can be corrected by sensing external environmental disturbances.

[0040] By constructing a fine-grained computation graph through runtime trajectory analysis, the modeling granularity is reduced from the macroscopic operator level to the microscopic kernel level, and filtering rules are used to eliminate low-level noise from heterogeneous hardware. This reasoning logic ensures that the model captures the physical essence of hardware execution rather than the logical description on the surface of the code, thus solving the cross-platform prediction inaccuracy problem caused by overly coarse representation in traditional methods and significantly improving the generalization accuracy of prediction results under different computing power environments.

[0041] S103. A nested cross-validation mechanism is adopted to match a corresponding predictor for each performance indicator in the preset performance indicator set.

[0042] The predictor includes a prediction model, a feature subset, and a hyperparameter combination; the nested cross-validation mechanism includes outer cross-validation and inner cross-validation; the outer cross-validation is used to determine the prediction model corresponding to each performance metric; the inner cross-validation is used to determine the feature subset and hyperparameter combination corresponding to each performance metric.

[0043] The preset performance metrics set includes performance metrics such as execution time in a single GPU environment, peak GPU utilization, average GPU utilization, peak VRAM usage, average VRAM usage, and CPU utilization.

[0044] In one feasible implementation: The preset model table is traversed, and the feature subsets and hyperparameter combinations corresponding to each performance index obtained by the inner cross-validation are used to train and optimize multiple candidate models in the preset model table on the training set, and the performance is evaluated on the test set. The prediction model corresponding to each performance index is determined based on the performance evaluation results.

[0045] like Figure 3 As shown, Figure 3A flowchart of nested cross-validation is presented. The selection process for the prediction model is illustrated using the outer five-fold cross-validation as an example, and the determination process of the feature subset and hyperparameter combination is illustrated using the inner three-fold cross-validation as an example. Specifically, the process of determining the prediction model through the outer five-fold cross-validation is as follows: The original training set is evenly divided into 5 folds. Iterations are performed with 4 folds as the training set and 1 fold as the test set. Each fold iterates through the predefined model table, as shown in Table 1, and initiates an inner-layer cross-validation process for each model to obtain the optimal feature subset and parameter combination.

[0046] Table 1

[0047] As shown in Table 1, for the Extreme Gradient Boosting (XGBoost) model, fixed hyperparameters are set to unify the experimental benchmark and there is no additional search space. Specifically, the histogram bin number max_bin is set to 256, the random seed random_state is set to 42, and the log output level verbosity is set to 0. For the RandomForest model, the hyperparameter search space includes: the search range for the number of decision trees n_estimators is [50, 100, 150]; the search range for the maximum depth of a single tree max_depth is [3, 5, 7, 10]; the search range for the minimum number of samples required for node split min_samples_split is [2, 5, 10]; the search range for the minimum number of samples required for leaf nodes min_samples_leaf is [2, 4, 8]; the search range for the maximum number of features to consider during splitting max_features is ['sqrt', 'log2', 0.7], where sqrt = the square root of the number of features, log2 represents taking the logarithm, and 0.7 represents taking 70% of the features; the bootstrap parameter is fixed to True; and the out-of-bag scoring parameter oob_score is fixed to True. For the GradientBoosting model, the search range for the number of decision trees n_estimators is [50, 100]; the search range for the learning rate learning_rate is [0.01, 0.03, 0.05]; the search range for the maximum depth of a single tree max_depth is [2, 3, 4]; the search range for the minimum number of samples required for node splitting min_samples_split is [5, 10]; the search range for the minimum number of samples required for leaf nodes min_samples_leaf is [2, 4]; the search range for the sample sampling ratio subsample is [0.7, 0.8, 0.9]; and the search range for the feature sampling ratio max_features is ['sqrt', 'log2'].For the Categorical Boosting (CatBoost) model, the search range for iterations is [50, 100]; the search range for learning rate is [0.01, 0.03, 0.05]; the search range for maximum tree depth is [3, 4, 6]; the search range for L2 regularization coefficient l2_leaf_reg is [1, 3, 5, 7]; the search range for Bayesian sampling temperature bagging_temperature is [0, 1]; the search range for feature scoring random strength is [0.1, 1]; the search range for numerical feature bin count border_count is [32, 64]; and the search range for leaf node value iterations is [5, 10]. For the Support Vector Regression (SVR) model, the search range for the penalty coefficient C is [0.1, 0.5, 1, 5]; the search range for the kernel coefficient gamma is ['scale', 'auto', 0.01, 0.001]; the search range for the kernel type kernel is ['rbf', 'linear', 'poly']; the search range for the polynomial kernel degree is [2, 3]; and the search range for the ε-insensitive loss bandwidth epsilon is [0.05, 0.1, 0.2]. For the K-Nearest Neighbors (KNN) model, the search range for the number of neighbors n_neighbors is [5, 7, 9, 11, 15]; the search range for the neighbor weights is ['uniform', 'distance']; the search range for the Minkowski distance order p is [1, 2]; the search range for the leaf node size of the tree is [20, 30, 40]; and the search range for the nearest neighbor search algorithm is ['auto', 'ball_tree', 'kd_tree'].For the Adaptive Boosting (AdaBoost) model, the search range for the number of decision trees, n_estimators, is [30, 50, 70]; the search range for the learning rate, learning_rate, is [0.01, 0.05, 0.1]; the search range for the loss function, loss, is ['linear', 'square', 'exponential']; and the base learners, estimators, are set to two fixed decision tree regressors: DecisionTreeRegressor(max_depth=2, random_state=42) and DecisionTreeRegressor(max_depth=3, random_state=42). For the Polynomial Ridge (PolyRidge) model, the polynomial order, poly_degree, is fixed at 2; the search range for the ridge regression regularization coefficient, ridge_alpha, is [0.1, 1.0, 10.0, 100.0]; and the search range for the solver algorithm, ridge_solver, is ['auto', 'svd'].

[0048] The training model is configured based on the results returned by the inner cross-validation process, and its performance is evaluated on the outer test set: coefficient of determination. And mean squared error (MSE), ultimately choosing the 50% average MSE with the lowest MSE and The highest-performing model, and its stability metrics are recorded. The formula for calculating MSE is as follows: ; ; Where k represents the cross-validation fold number; Indicates the j-th fold value.

[0049] In another feasible implementation: By employing multiple dimensionality reduction strategies, the initial feature matrix is ​​subjected to dimensionality reduction processing based on multiple performance indicators in a preset set of performance indicators, thereby obtaining a feature subset of each performance indicator under each dimensionality reduction strategy.

[0050] Based on preset screening criteria, the feature subsets of each performance indicator under various dimensionality reduction strategies are screened to determine the feature subset corresponding to each performance indicator.

[0051] Hyperparameter optimization is performed based on the feature subset corresponding to each performance index to obtain the hyperparameter combination corresponding to each performance index.

[0052] There are no restrictions on the preset filtering conditions here. For example, for a feature subset, select features that appear more than or equal to 2 times in the 3-fold analysis.

[0053] In one feasible implementation, the multiple dimensionality reduction strategies include a three-level progressive dimensionality reduction strategy. Using these strategies, the initial feature matrix is ​​dimensionality-reduced based on multiple performance indicators from a preset set of performance indicators, resulting in a feature subset for each performance indicator under each dimensionality reduction strategy. This includes: Using a three-level progressive dimensionality reduction strategy, the initial feature matrix is ​​dimensionality reduced according to multiple performance indicators in a preset performance indicator set, to obtain a feature subset of each performance indicator under the three-level progressive dimensionality reduction strategy; the three-level progressive dimensionality reduction strategy includes feature selection, feature compression and collinearity elimination in sequence.

[0054] In one feasible implementation, a three-level progressive dimensionality reduction strategy is used to reduce the dimensionality of the initial feature matrix according to multiple performance indicators in a preset set of performance indicators, thereby obtaining a feature subset for each performance indicator under the three-level progressive dimensionality reduction strategy, including: The initial feature matrix is ​​saliency-filtered based on the importance of random forest features to obtain the first-level dimensionality reduction result.

[0055] The first-level dimensionality reduction result is sparsely compressed using L1 regularization, and then optimized according to a preset objective function to obtain the second-level dimensionality reduction result.

[0056] The multicollinearity among the features of the second-level dimensionality reduction result is quantified by the variance inflation factor, and features exceeding a preset threshold are removed based on the quantification result to obtain the third-level dimensionality reduction result.

[0057] There are no restrictions on the preset threshold here, for example, 10.

[0058] The three-stage progressive dimensionality reduction strategy consists of Boruta significance test, LASSO (Least Absolute Shrinkage and Selection Operator) sparsity compression, and Variance Inflation Factor (VIF) collinearity removal. The Boruta stage uses shadow feature comparison to filter out random noise that is irrelevant to the target index. The LASSO stage uses L1 regularization to achieve sparsity of the feature space and retain the core dimensions with high contribution. The VIF stage is specifically designed to remove multicollinearity to ensure the independence and stability of the feature subset.

[0059] In another feasible implementation, the multiple dimensionality reduction strategies include a recursive feature elimination strategy. Utilizing these strategies, the initial feature matrix is ​​dimensionality-reduced based on multiple performance indicators from a preset set of performance indicators, resulting in a feature subset for each performance indicator under each dimensionality reduction strategy. This includes: Using a recursive feature elimination strategy, the initial feature matrix is ​​dimensionality-reduced based on multiple performance indicators in a preset performance indicator set to obtain a feature subset for each performance indicator under the recursive feature elimination strategy. The recursive feature elimination strategy is used to perform multiple iterations of filtering on the initial feature matrix based on feature importance scores. In each iteration, the feature with the lowest current contribution is removed. Through multi-fold cross-validation, negative mean square error is used as the evaluation index to obtain the feature subset with the best generalization performance.

[0060] like Figure 3 As shown, Figure 3 This document presents a flowchart of a nested cross-validation method. Taking the outer five-fold cross-validation as an example, it illustrates the selection process of the prediction model. The inner three-fold cross-validation method is used as an example to illustrate the determination process of the feature subset and hyperparameter combination. Specifically, the process of determining the feature subset and hyperparameter combination through the inner three-fold cross-validation is as follows: The outer training set is further divided into three folds, generating the inner training and validation sets in a 2:1 ratio. Considering that the initial feature matrix is ​​prone to overfitting and may contain noisy features, redundant information, and multicollinearity, a dimensionality reduction strategy is used to achieve accurate feature selection and optimization.

[0061] The Recursive Feature Elimination (RFE) strategy is used to iteratively remove features that contribute the least to the model. It uses 5-fold cross-validation and negative mean squared error to evaluate and select the feature subset with the best generalization performance.

[0062] The three-level progressive dimensionality reduction strategy involves feature selection, feature compression, and collinearity removal. Specifically, significant features are selected based on the random forest importance Z-score using a significance threshold of 0.05.

[0063] set up Where d is the number of features, and the shadow feature matrix is ​​generated by shuffling each column. ,satisfy: .

[0064] Train a random forest with T trees. For the t-th tree, calculate the initial features. .

[0065] For initial features Calculate importance mean and standard deviation The formulas are as follows:

[0066] ; in, This represents the importance score of the j-th initial feature in the t-th iteration.

[0067] The Z-score is used to measure the significance of the initial features relative to the shadow features, as shown in the following formula: ; in, This represents the Z-score value of the j-th initial feature. The larger the value, the more important the initial feature is than the random noise, and the more effective the feature is. This represents the importance score of the j-th initial feature. LASSO feature compression uses L1 regularization to compress the feature space and optimizes the objective function, which is shown in the following formula: ; in, The solution process aims to find the parameter vector that minimizes the subsequent function value. n represents the total number of training samples; This represents the true target value of the i-th sample; This represents the feature vector corresponding to the i-th sample; This represents the parameter vector to be solved in the model; p represents the total number of features. Represents the regularization coefficient. The larger the value, the heavier the penalty for the model's complexity coefficient, and more parameters will be compressed to 0, thereby achieving feature sparsity.

[0068] Features with a VIF greater than 10 are removed. The formula for calculating VIF is as follows: ; in, Let represent the variance inflation factor of the j-th feature; This represents the square of the multiple correlation coefficient obtained by regressing the j-th feature as the dependent variable against all other independent variables. It measures the proportion of variation that the j-th feature can explain by the other features. The closer the value is to 1, the stronger the linear correlation between the feature and other features, and the more severe the collinearity.

[0069] In the reduced feature space, a grid search is performed using the inner training set to find the optimal hyperparameters of the model at that fold. The inner validation set R² is used as the optimization objective, and the hyperparameter combination corresponding to the highest R² in the 3 folds is selected as the optimal hyperparameter combination.

[0070] The determined optimal feature subset and optimal hyperparameter combination are returned to the outer layer. Through the above nested cross-validation process, a unique optimal prediction configuration is determined for each performance metric, specifically including the most suitable prediction model, the optimal feature subset, and the optimal hyperparameter combination.

[0071] After the configuration is determined, each model is retrained based on the complete training set to obtain the final predictor for each performance metric.

[0072] This nested cross-validation mechanism, targeting multiple performance metrics, employs an outer loop for globally fair model selection and an inner loop for feature subset locking and hyperparameter search specific to each metric, enabling one-stop prediction of multiple performance indicators. By combining nested cross-validation with a three-level progressive dimensionality reduction approach, the uncertainties of manual tuning are eliminated, allowing performance prediction results to reflect the resource competition status in a dynamic cluster in real time. This provides scientific and robust decision support for the lean orchestration of computing resources and the avoidance of systemic collapse.

[0073] S104. Based on the predictor that corresponds one-to-one with multiple performance indicators in the preset performance indicator set, perform runtime performance prediction on the task to be processed to obtain the performance prediction result.

[0074] These predictors performed performance predictions on independent test sets, yielding a set of performance prediction results. This set of results can be transformed into visual performance profile charts or JSON reports, which can be sent to end users or directly fed back to the operator scheduling system. Users can then obtain information about the expected performance of tasks in the current environment, enabling precise orchestration of computing resources.

[0075] The deep learning training task runtime performance determination method provided in this application embodiment obtains the original runtime events of the task to be processed, and constructs a fine-grained computation graph based on the execution trajectory sequence corresponding to the original runtime events; extracts multi-dimensional features of the fine-grained computation graph to obtain an initial feature matrix; the multi-dimensional features include node attribute features, edge attribute features, environment perception features, hyperparameter features, and graph topology features; a nested cross-validation mechanism is used to match a corresponding predictor for each performance indicator in a preset performance indicator set; the predictor includes a prediction model, a feature subset, and a hyperparameter combination; the nested cross-validation mechanism includes outer cross-validation and inner cross-validation; the outer cross-validation is used to determine the prediction model corresponding to each performance indicator; the inner cross-validation is used to determine the feature subset and hyperparameter combination corresponding to each performance indicator; and the runtime performance of the task to be processed is predicted based on the predictors that correspond one-to-one with the multiple performance indicators in the preset performance indicator set to obtain the performance prediction result.

[0076] The above method constructs a fine-grained computation graph based on the execution trajectory sequence of the original runtime events. This fully preserves the underlying runtime details, such as operator execution order, data dependencies, memory read / write, and communication patterns, avoiding prediction bias caused by information loss and laying a high-quality feature foundation for accurate runtime performance prediction. Extracting multi-dimensional features from the fine-grained computation graph allows for a more comprehensive capture of the impact of various factors on runtime performance, significantly reducing prediction inaccuracies caused by incomplete features and improving the reliability and accuracy of runtime performance prediction results. A nested cross-validation mechanism is employed, using outer cross-validation to optimize the prediction model and inner cross-validation to select the optimal feature subset and hyperparameter combination. This achieves fine-grained adaptation between each performance metric and the predictor, significantly improving the accuracy of the overall performance prediction results. Joint prediction based on dedicated predictors corresponding one-to-one with multiple performance metrics accurately captures the changing trends of different performance metrics, avoiding mutual interference between them. Even in complex deep learning training scenarios, it maintains high prediction accuracy, making the final performance prediction results closer to real runtime performance.

[0077] Based on the method for determining the runtime performance of a deep learning training task as described in the preceding embodiments, this application also provides a device for determining the runtime performance of a deep learning training task. Figure 4 This is a schematic diagram of the device. Figure 4 As shown, the device for determining the runtime performance of a deep learning training task includes: Graph construction module 401 is used to obtain the original runtime events of the task to be processed, and construct a fine-grained computation graph based on the execution trajectory sequence corresponding to the original runtime events. The multi-dimensional feature extraction module 402 is used to extract multi-dimensional features of the fine-grained computation graph to obtain an initial feature matrix; the multi-dimensional features include node attribute features, edge attribute features, environment perception features, hyperparameter features, and graph topology features.

[0078] The predictor determination module 403 is used to match a corresponding predictor for each performance indicator in a preset set of performance indicators using a nested cross-validation mechanism; the predictor includes a prediction model, a feature subset, and a hyperparameter combination; the nested cross-validation mechanism includes outer cross-validation and inner cross-validation; the outer cross-validation is used to determine the prediction model corresponding to each performance indicator; the inner cross-validation is used to determine the feature subset and hyperparameter combination corresponding to each performance indicator.

[0079] The performance prediction module 404 is used to perform runtime performance prediction on the task to be processed based on a predictor that corresponds one-to-one with multiple performance indicators in the preset performance indicator set, and obtain the performance prediction result.

[0080] Optionally, the predictor determination module is used for: The preset model table is traversed, and the feature subsets and hyperparameter combinations corresponding to each performance index obtained by the inner cross-validation are used to train and optimize multiple candidate models in the preset model table on the training set, and the performance is evaluated on the test set. The prediction model corresponding to each performance index is determined based on the performance evaluation results.

[0081] Optionally, the predictor determination module includes: The dimensionality reduction processing unit is used to perform dimensionality reduction processing on the initial feature matrix according to multiple performance indicators in a preset performance indicator set using various dimensionality reduction strategies, so as to obtain the feature subset of each performance indicator under each dimensionality reduction strategy.

[0082] The feature subset determination unit is used to filter the feature subsets of each performance indicator under multiple dimensionality reduction strategies according to preset screening conditions, and determine the feature subset corresponding to each performance indicator.

[0083] The hyperparameter combination determination unit performs hyperparameter optimization based on the feature subset corresponding to each performance index to obtain the hyperparameter combination corresponding to each performance index.

[0084] Optionally, various dimensionality reduction strategies include a three-level progressive dimensionality reduction strategy, and the dimensionality reduction processing unit is specifically used for: Using a three-level progressive dimensionality reduction strategy, the initial feature matrix is ​​dimensionality reduced according to multiple performance indicators in a preset performance indicator set, to obtain a feature subset of each performance indicator under the three-level progressive dimensionality reduction strategy; the three-level progressive dimensionality reduction strategy includes feature selection, feature compression and collinearity elimination in sequence.

[0085] Optionally, the three-level progressive dimensionality reduction strategy is used to reduce the dimensionality of the initial feature matrix according to multiple performance indicators in a preset performance indicator set, to obtain a feature subset of each performance indicator under the three-level progressive dimensionality reduction strategy, including: The initial feature matrix is ​​saliency-filtered based on the importance of random forest features to obtain the first-level dimensionality reduction result.

[0086] The first-level dimensionality reduction result is sparsely compressed using L1 regularization, and then optimized according to a preset objective function to obtain the second-level dimensionality reduction result.

[0087] The multicollinearity among the features of the second-level dimensionality reduction result is quantified by the variance inflation factor, and features exceeding a preset threshold are removed based on the quantification result to obtain the third-level dimensionality reduction result.

[0088] Optionally, various dimensionality reduction strategies include recursive feature elimination strategies, and the dimensionality reduction processing unit is specifically used for: Using a recursive feature elimination strategy, the initial feature matrix is ​​dimensionality-reduced based on multiple performance indicators in a preset performance indicator set to obtain a feature subset for each performance indicator under the recursive feature elimination strategy. The recursive feature elimination strategy is used to perform multiple iterations of filtering on the initial feature matrix based on feature importance scores. In each iteration, the feature with the lowest current contribution is removed. Through multi-fold cross-validation, negative mean square error is used as the evaluation index to obtain the feature subset with the best generalization performance.

[0089] Optionally, the multi-dimensional features include graph topological features, and the multi-dimensional feature extraction module is specifically used for: Based on the scale of the fine-grained computation graph, a corresponding graph topology feature extraction strategy is determined; the graph topology feature extraction strategy includes feature analysis depth and sampling ratio.

[0090] The graph topology features of the fine-grained computation graph are extracted using the graph topology feature extraction strategy to obtain the graph topology feature extraction results.

[0091] Optionally, the graph building module is specifically used for: The original runtime events are filtered using preset rules to obtain filtered runtime events.

[0092] The filtered runtime events are constructed based on the call stack and timestamp information to obtain a fine-grained computation graph.

[0093] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for determining the runtime performance of a deep learning training task as described in any of the method embodiments.

[0094] Furthermore, this application embodiment also provides a processor for running a computer program, wherein the program executes the deep learning training task runtime performance determination method as described in any of the aforementioned method embodiments.

[0095] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. The components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment solution according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0096] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for determining runtime performance of a deep learning training task, characterized in that, include: Obtain the original runtime events of the task to be processed, and construct a fine-grained computation graph based on the execution trajectory sequence corresponding to the original runtime events; Extract the multi-dimensional features of the fine-grained computation graph to obtain an initial feature matrix; the multi-dimensional features include node attribute features, edge attribute features, environment perception features, hyperparameter features, and graph topology features; A nested cross-validation mechanism is employed to match a corresponding predictor for each performance indicator in a preset set of performance indicators. The predictor includes a prediction model, a feature subset, and a hyperparameter combination. The nested cross-validation mechanism includes an outer cross-validation layer and an inner cross-validation layer. The outer cross-validation layer is used to determine the prediction model corresponding to each performance indicator. The inner cross-validation layer is used to determine the feature subset and hyperparameter combination corresponding to each performance indicator. The runtime performance of the task to be processed is predicted based on a predictor that corresponds one-to-one with multiple performance indicators in the preset performance indicator set, and the performance prediction result is obtained.

2. The method according to claim 1, characterized in that, The method employs a nested cross-validation mechanism to match a corresponding predictor for each performance metric in a preset set of performance metrics, including: The preset model table is traversed, and the feature subsets and hyperparameter combinations corresponding to each performance index obtained by the inner cross-validation are used to train and optimize multiple candidate models in the preset model table on the training set, and the performance is evaluated on the test set. The prediction model corresponding to each performance index is determined based on the performance evaluation results.

3. The method according to claim 1, characterized in that, The method employs a nested cross-validation mechanism to match a corresponding predictor for each performance metric in a preset set of performance metrics, including: Using multiple dimensionality reduction strategies, the initial feature matrix is ​​dimensionality reduced according to multiple performance indicators in a preset set of performance indicators, to obtain the feature subset of each performance indicator under each dimensionality reduction strategy; Based on preset screening criteria, the feature subsets of each performance indicator under multiple dimensionality reduction strategies are screened to determine the feature subset corresponding to each performance indicator; Hyperparameter optimization is performed based on the feature subset corresponding to each performance index to obtain the hyperparameter combination corresponding to each performance index.

4. The method according to claim 3, characterized in that, The multiple dimensionality reduction strategies include a three-level progressive dimensionality reduction strategy. The initial feature matrix is ​​dimensionality-reduced using these strategies based on multiple performance indicators from a preset set of performance indicators, resulting in a feature subset for each performance indicator under each dimensionality reduction strategy. This includes: Using a three-level progressive dimensionality reduction strategy, the initial feature matrix is ​​dimensionality reduced according to multiple performance indicators in a preset performance indicator set, to obtain a feature subset of each performance indicator under the three-level progressive dimensionality reduction strategy; the three-level progressive dimensionality reduction strategy includes feature selection, feature compression and collinearity elimination in sequence.

5. The method according to claim 4, characterized in that, The three-level progressive dimensionality reduction strategy is used to reduce the dimensionality of the initial feature matrix according to multiple performance indicators in a preset set of performance indicators, obtaining a feature subset for each performance indicator under the three-level progressive dimensionality reduction strategy, including: The initial feature matrix is ​​saliency-based by random forest feature importance to obtain the first-level dimensionality reduction result; The first-level dimensionality reduction result is sparsely compressed by L1 regularization, and then optimized according to the preset objective function to obtain the second-level dimensionality reduction result. The multicollinearity among the features of the second-level dimensionality reduction result is quantified by the variance inflation factor, and features exceeding a preset threshold are removed based on the quantification result to obtain the third-level dimensionality reduction result.

6. The method according to claim 3, characterized in that, The multiple dimensionality reduction strategies include a recursive feature elimination strategy. The initial feature matrix is ​​dimensionality-reduced using these strategies based on multiple performance indicators from a preset set of performance indicators, resulting in a feature subset for each performance indicator under each dimensionality reduction strategy. This includes: Using a recursive feature elimination strategy, the initial feature matrix is ​​dimensionality-reduced based on multiple performance indicators in a preset performance indicator set to obtain a feature subset for each performance indicator under the recursive feature elimination strategy. The recursive feature elimination strategy is used to perform multiple iterations of filtering on the initial feature matrix based on feature importance scores. In each iteration, the feature with the lowest current contribution is removed. Through multi-fold cross-validation, negative mean square error is used as the evaluation index to obtain the feature subset with the best generalization performance.

7. The method according to claim 1, characterized in that, The multi-dimensional features include graph topological features. Extracting the multi-dimensional features of the fine-grained computation graph to obtain an initial feature matrix includes: Based on the scale of the fine-grained computation graph, a corresponding graph topology feature extraction strategy is determined; the graph topology feature extraction strategy includes feature analysis depth and sampling ratio. The graph topology features of the fine-grained computation graph are extracted using the graph topology feature extraction strategy to obtain the graph topology feature extraction results.

8. The method according to claim 1, characterized in that, The construction of a fine-grained computation graph based on the execution trajectory sequence corresponding to the original runtime events includes: The original runtime events are filtered using preset rules to obtain filtered runtime events; The filtered runtime events are constructed based on the call stack and timestamp information to obtain a fine-grained computation graph.

9. A device for determining the runtime performance of a deep learning training task, characterized in that, include: The graph construction module is used to obtain the original runtime events of the task to be processed, and to construct a fine-grained computation graph based on the execution trajectory sequence corresponding to the original runtime events. A multi-dimensional feature extraction module is used to extract multi-dimensional features of the fine-grained computation graph to obtain an initial feature matrix; the multi-dimensional features include node attribute features, edge attribute features, environment perception features, hyperparameter features, and graph topology features; A predictor determination module is used to match a corresponding predictor for each performance indicator in a preset set of performance indicators using a nested cross-validation mechanism. The predictor includes a prediction model, a feature subset, and a hyperparameter combination. The nested cross-validation mechanism includes outer cross-validation and inner cross-validation. The outer cross-validation is used to determine the prediction model corresponding to each performance indicator. The inner cross-validation is used to determine the feature subset and hyperparameter combination corresponding to each performance indicator. The performance prediction module is used to perform runtime performance prediction on the task to be processed based on a predictor that corresponds one-to-one with multiple performance indicators in the preset performance indicator set, and obtain the performance prediction result.

10. A processor, characterized in that, Used to run a computer program, which, when running, performs the runtime performance determination method for a deep learning training task as described in any one of claims 1-7.

Images