Distributed machine learning training performance prediction method and device

Abstract

The invention provides a distributed machine learning training performance prediction method and device. The method comprises the following steps: selecting a performance prediction target; determining a screened characteristic parameter set according to performance prediction; acquiring training performance parameters and determining an original data set according to the plurality of screened characteristic parameters; according to the configuration parameters and the training performance parameters, constructing a source domain performance model by using a random forest algorithm, and training the source domain performance model by using part of the original data set to obtain a trained source domain performance model; constructing a target cluster performance prediction model by using a support vector regression algorithm according to a part of migration samples of the target cluster, the trained source domain performance model and the migration model; and inputting the source cluster and the target cluster into the trained target cluster performance prediction model to realize cross-cluster migration from the performance of the source cluster to the performance of the target cluster, and predicting performance parameters of the cross-cluster migration. Therefore, the prediction accuracy is relatively high, the prediction efficiency is relatively high, and the performance prediction cost is relatively low.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed machine learning training and configurable software system performance prediction, and particularly relates to a distributed machine learning training performance prediction method and device. BACKGROUND

[0002] With the advent of the big data era and the surge of the artificial intelligence wave, the demand for machine learning training tasks is becoming increasingly strong, and therefore distributed machine learning has become one of the current popular research directions. Due to the explosive growth of data quantity and the increase of machine learning model complexity, the traditional single-machine single-card training method cannot meet the resource demand and time cost demand in the machine learning training process, and therefore the application of distributed technology to the machine learning training process, i.e. distributed machine learning (DML) training, has attracted widespread attention from the academic and industrial circles.

[0003] At present, there are two schemes for distributed machine learning training performance prediction. One is an efficient distributed machine learning training system based on a serverless architecture, which symbolically models the performance prediction model among parameters, training delay, model accuracy and economic cost by analyzing the characteristics of communication and aggregation of the serverless architecture, so as to realize the optimization of the configuration parameters, training cost and performance of the distributed machine learning training system. However, due to the complexity of the analysis of the internal communication and aggregation principle of the distributed machine learning system, it is difficult to accurately fit, and there is a problem of dynamic change, which makes the prediction accuracy low. The other is a device performance statistical data processing method and system, which is suitable for scenarios where device performance historical data has time series characteristics and discrete working state classification values are used as prediction targets. However, using a long short-term memory network for device performance prediction requires frequent forward propagation and back propagation to calculate gradients, involves a large number of matrix operations, and the performance prediction cost of the performance prediction model is high. Since the performance prediction method constructed using the long short-term memory network needs to capture performance data in a long time period and detect its dynamic change trend, a large-scale data set is required to support prediction, and the model is relatively complex, a large number of matrix operations occupy computing resources and reduce prediction speed, which makes the prediction efficiency low in both time and space. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a distributed machine learning training performance prediction method and device, which solves the problems of low prediction accuracy, high performance prediction cost and low prediction efficiency.

[0005] To solve the above technical problems, the embodiments of the present application provide the following technical solutions: The first aspect of the present application provides a distributed machine learning training performance prediction method, comprising: select a performance prediction target according to load characteristics of distributed machine learning training and performance prediction demand in a target scenario; perform filtering on all feature parameters according to the performance prediction target to obtain a filtered feature parameter set, the filtered feature parameter set including a plurality of filtered feature parameters; collect training performance parameters in the distributed machine learning system by using a random sampling method according to the plurality of filtered feature parameters, and use the training performance parameters and corresponding configuration parameters as an original data set; construct a source domain performance model by using a random forest algorithm according to the configuration parameters and the training performance parameters, and train the source domain performance model by using part of the original data set to obtain a trained source domain performance model; construct a target cluster performance prediction model by using a support vector regression algorithm according to part of migration samples of the target cluster, the trained source domain performance model and a migration model; input the source cluster and the target cluster into the trained target cluster performance prediction model to realize cross-cluster migration of performance of the source cluster to performance of the target cluster by using the trained target cluster performance prediction model, and predict a performance parameter of the cross-cluster migration.

[0006] The second aspect of the application provides a distributed machine learning training performance prediction device, including: The selection module is configured to select a performance prediction target according to load characteristics of distributed machine learning training and performance prediction demand in a target scenario; The filtering module is configured to perform filtering on all feature parameters according to the performance prediction target to obtain a filtered feature parameter set, the filtered feature parameter set including a plurality of filtered feature parameters; The collection module is configured to collect training performance parameters in the distributed machine learning system by using a random sampling method according to the plurality of filtered feature parameters, and use the training performance parameters and corresponding configuration parameters as an original data set; The first construction module is configured to construct a source domain performance model by using a random forest algorithm according to the configuration parameters and the training performance parameters, and train the source domain performance model by using part of the original data set to obtain a trained source domain performance model; The second construction module is configured to construct a target cluster performance prediction model by using a support vector regression algorithm according to part of migration samples of the target cluster, the trained source domain performance model and a migration model; The migration module is configured to input the source cluster and the target cluster into the trained target cluster performance prediction model to realize cross-cluster migration of performance of the source cluster to performance of the target cluster by using the trained target cluster performance prediction model, and predict a performance parameter of the cross-cluster migration.

[0007] The third aspect of the present application provides an electronic device, comprising: at least one processor; and at least one memory connected with the processor through a bus; wherein the processor and the memory complete mutual communication through the bus; the processor is used to call program instructions in the memory to execute the distributed machine learning training performance prediction method of the first aspect or any one of the optional embodiments of the first aspect.

[0008] Compared with the prior art, the distributed machine learning training performance prediction method and device provided by the present application selects a performance prediction target according to the load characteristics of the distributed machine learning training and the performance prediction requirements in the target scenario, filters all feature parameters according to the performance prediction target to obtain a set of filtered feature parameters, collects training performance parameters in the distributed machine learning system by using a random sampling method according to the plurality of filtered feature parameters, and takes the training performance parameters and the corresponding configuration parameters as an original data set, constructs a source domain performance model by using a random forest algorithm according to the configuration parameters and the training performance parameters, trains the source domain performance model by using part of the original data set to obtain a trained source domain performance model, constructs a target cluster performance prediction model by using a support vector regression algorithm according to part of the migration samples of the target cluster, the trained source domain performance model and the migration model, inputs the source cluster and the target cluster into the trained target cluster performance prediction model to realize the cross-cluster migration of the performance of the source cluster to the performance of the target cluster by using the trained target cluster performance prediction model, and predicts the performance parameters of the cross-cluster migration. In this way, filtering the parallel configuration parameters can reduce the number of features, the machine learning based on the random forest algorithm constructs the source domain performance model, the random forest algorithm is used in the field of distributed machine learning performance prediction, and the target cluster performance prediction model is determined by using the transfer learning method to predict the performance parameters of the cross-cluster migration, so that the prediction accuracy is higher and the prediction efficiency is higher. The sampling cost is greatly reduced by the cross-cluster migration method, so that the performance prediction cost is lower. BRIEF DESCRIPTION OF DRAWINGS

[0009] The above and other objects, features and advantages of the present application exemplary embodiments will be more apparent from the following detailed description read in conjunction with the accompanying drawings, in which several embodiments of the present application are shown by way of example, and wherein like reference numerals refer to like elements throughout. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the present application. In the drawings: Figure 1 A flowchart of the distributed machine learning training performance prediction method is schematically shown; Figure 2 A round training time model evaluation schematic diagram is schematically shown; Figure 3 A video memory occupation prediction model evaluation schematic diagram is schematically shown; Figure 4 a migration model round training time evaluation schematic diagram is shown schematically; Figure 5 a migration model GPU memory occupation time evaluation schematic diagram is shown schematically; Figure 6 a structural diagram of a distributed machine learning training performance prediction device is shown schematically; Figure 7 a structural diagram of an electronic device is shown schematically. DETAILED DESCRIPTION

[0010] Exemplary embodiments of the present application will be described in detail with reference to the drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0011] It should be noted that: unless otherwise specified, the technical terms or scientific terms used in the present application should be understood as the usual meaning understood by those skilled in the art to which the present application belongs.

[0012] The method in the embodiments of the present application will be described in detail below.

[0013] Figure 1 a flowchart of the distributed machine learning training performance prediction method in the embodiments of the present application is shown schematically, referring to Figure 1 As shown, the distributed machine learning training performance prediction method can include: S101, selecting a performance prediction target according to the load characteristics of distributed machine learning training and the performance prediction requirement in the target scenario.

[0014] The performance prediction target includes training loss, training accuracy, test loss, test accuracy, round training time, GPU occupation, GPU memory occupation and memory occupation.

[0015] The target scenario is any actual scenario.

[0016] Specifically, in the distributed machine learning training system, there are different indicators to evaluate the performance of training, including training loss, training accuracy, test loss, test accuracy, round training time, GPU occupation, GPU memory occupation, memory occupation, etc. According to the load characteristics of distributed machine learning training, combined with the performance prediction requirement in the actual scenario, the performance indicator set to be concerned for performance prediction is selected, i.e. the set corresponding to the performance prediction target is , is the total number of performance prediction targets to be predicted, is a specific single performance index, wherein, is the performance prediction target.

[0017] S102, screening all feature parameters according to the performance prediction target to obtain a screened feature parameter set.

[0018] The screened feature parameter set includes a plurality of screened feature parameters.

[0019] The selection of feature parameters aims to analyze the function of the distributed training commonly used feature parameters in the training process according to the performance prediction target determined in step S101, and screen out the configuration parameter set that may affect the aforementioned performance prediction target, which is processed as the input feature of the performance prediction model, including two sub-steps of redundant feature elimination and dependent feature decoupling.

[0020] Specifically, screening all feature parameters according to the performance prediction target to obtain a screened feature parameter set, including: Step A1: Among all feature parameters of the distributed machine learning training performance file, the initial configuration parameters corresponding to the performance prediction target are determined.

[0021] Step A1 is a step of redundant feature elimination, and step A1 is used to screen out parallel configuration features that have little effect on the performance of distributed machine learning training, so as to reduce the computational complexity of the algorithm and improve the interpretability of the model. Specifically, by consulting the relevant materials of the distributed machine learning training performance configuration parameter table, that is, by consulting the distributed machine learning training performance file, the distributed machine learning training configuration parameters that may be related to the performance prediction target are preliminarily listed, the control effect of a single parameter in the distributed machine learning training is analyzed, and the redundant parallel features that are obviously irrelevant to performance are eliminated.

[0022] Step A2: According to the principle of multi-dimensional parallel training of distributed machine learning, the dependency relationship of the features in the initial configuration parameters is determined, and the initial configuration parameters corresponding to the preset dependency relationship in the dependency relationship are decoupled to obtain a screened feature parameter set.

[0023] The dependency relationship includes a calculation relationship and a decision relationship.

[0024] Step A2 is a step of dependent feature decoupling. The preset dependency relationship is a relationship that obviously produces dependency.

[0025] Step A2 is used to reduce the dependence between feature parameters, eliminate potential constraints of algorithm input and reduce the number of features. Specifically, according to the principle of multi-dimensional parallel training of distributed machine learning, the dependence between features in the initial configuration parameters is analyzed, the set corresponding to the initial configuration parameters that obviously produce dependence is decoupled, and the minimum dependence set is retained to reduce the number of input features. Specifically, the dependence includes a calculation relationship and a decision relationship. The calculation relationship shows the relationship between features in a mathematical formula, such as the global thread number being equal to the product of the data parallel scale and the pipeline parallel scale. The decision relationship implicitly models the functional relationship between feature values in a black box manner. This step A2 should ensure that the decoupled features, i.e., the filtered feature parameter set, can be calculated from the retained feature values to obtain the minimum core parameter set for performance prediction. Specifically, the dependent feature decoupling is implemented through the following steps A21 to A24: Specifically, step A2 includes: Step A21: According to the principle of multi-dimensional parallel training of distributed machine learning, determine the calculation relationship and decision relationship of the features in the initial configuration parameters.

[0026] Step A22: Construct a directed graph with the initial configuration parameters as vertices and the calculation relationship and decision relationship as edges.

[0027] The directed graph includes multiple target connected components. Each weakly connected component is a directed acyclic graph. The target connected component is a weakly connected component, which is a component after ignoring the direction of all edges.

[0028] Step A23: Divide each target connected component into a cluster.

[0029] Step A24: Traverse each cluster and find vertices with a preset in-degree, and aggregate the vertices with the preset in-degree into a filtered feature parameter set.

[0030] The preset in-degree is 0, traverse each cluster and find vertices with an in-degree of 0, and aggregate them into a filtered feature parameter set.

[0031] S103, according to the plurality of filtered feature parameters, using a random sampling method to collect training performance parameters in a distributed machine learning system, and taking the training performance parameters and corresponding configuration parameters as an original data set.

[0032] The collection of training performance parameters includes two sub-steps of configuration extraction and performance sampling execution.

[0033] Specifically, according to the plurality of filtered feature parameters, using a random sampling method to collect training performance parameters in a distributed machine learning system, and taking the training performance parameters and corresponding configuration parameters as an original data set, including: Step B1: According to the random sampling method, a value is extracted from the value range corresponding to each of the plurality of screened feature parameters to generate a configuration file.

[0034] Step B1 is a sub-step of configuration extraction. Step B1 ensures the continuity of the experiment, and uses the random sampling method to extract a plurality of to-be-executed configurations in the feature space in advance, and generates representative configurations to reduce the scale of the experiment.

[0035] The set corresponding to the plurality of screened feature parameters is , and the screened feature parameter, i.e., the configuration parameter , corresponds to a value range , wherein is the value of the th configuration parameter ; according to the random sampling method, a value is randomly extracted from the value range corresponding to each configuration parameter, i.e. , wherein is the value of the th configuration parameter, is a random value function, and the function of the random value function is to randomly extract an element from a given set; after the random extraction of each configuration parameter is completed, all values are integrated as a row of to-be-executed configurations, and the to-be-executed configurations are written into the configuration file on the disk through file operation, wherein is the th value.

[0036] Step B2: Each row of features in the configuration file is read and decoded in turn, and the value of the corresponding configuration parameter is set in the distributed machine learning system based on the decoded features, to realize the training of the distributed machine learning, and obtain the training performance parameter.

[0037] Step B2 and step B3 are performance sampling execution sub-steps.

[0038] In the distributed machine learning system, setting the value of the corresponding configuration parameter is actually executing the distributed machine learning training process, obtaining the actual training performance under each to-be-sampled configuration, and obtaining the training performance parameter.

[0039] Step B3: The training performance parameter and the configuration parameter are used as the original data set.

[0040] The timestamp at the beginning of the experiment is used as the file name label, and the configuration parameter and the training performance parameter, i.e., the actual training performance obtained by sampling, are stored in the data file. All data files are arranged as a directory as the original data set.

[0041] S104. Based on the configuration parameters and training performance parameters, construct the source domain performance model using the random forest algorithm, and train the source domain performance model using a portion of the original dataset to obtain the trained source domain performance model.

[0042] Specifically, based on the configuration parameters and training performance parameters, a source domain performance model is constructed using the random forest algorithm, and this model is trained using a portion of the original dataset to obtain the trained source domain performance model, including: Step C1: Associate the configuration parameters and training performance parameters by execution timestamp, and delete any configuration parameters that have not been executed, to obtain vector tuples.

[0043] The vector tuple includes a configuration parameter vector and a performance metric vector.

[0044] By using execution timestamps, configuration parameters and training performance parameters are associated, which means establishing a mapping relationship between configuration parameters and training performance parameters.

[0045] Vector tuples are represented as ,in, To configure the parameter vector, The performance metric vector is the vector of interest for performance prediction. The vector tuples are converted into a data table, serialized, and exported as a performance data file, which is then stored on disk.

[0046] Step C2: Use the random forest algorithm to construct a source domain performance model for each component in the performance index vector.

[0047] Specifically, import the performance data file and read all configuration parameter vectors and vector tuples. For performance index vectors Each component (i.e., each performance metric) in the equation. The source domain performance model of this component is constructed using the random forest algorithm. ,in, Representing the Each component The predicted value, This represents a performance prediction model learned through the random forest algorithm. This represents a vector of configuration parameters.

[0048] Specifically, the source domain performance model of this component is constructed using the random forest algorithm, following these steps: generating root nodes; randomly sampling samples; randomly extracting features; and calculating the Gini index of the nodes according to the expression for the Gini index. Nodes are split based on the Gini index of each node, and single decision trees are constructed; among them, The probability distribution representing the sample classification. a total number of classes representing the samples, a first a frequency of the class samples appearing in the node sample set; generating T a decision tree to combine into a source domain performance model, wherein T a size of the decision tree, and an average of outputs of all decision trees as an output of the source domain performance model.

[0049] Step C3: training the source domain performance model using part of the original data set to obtain a trained source domain performance model.

[0050] Specifically, the obtained distributed machine learning training performance original data set is randomly divided into an initial training set and a test set, and each of the two data sets includes multiple configuration parameters and training performance parameters for training.

[0051] Using a 10-fold cross-validation method on the initial training set, the initial training set is evenly divided into 10 parts, and each time 9 parts are selected as a training set to train the source domain performance model, and 1 part is selected as a validation set to optimize the hyperparameters of the source domain performance model. Specifically, dividing the training set and the validation set helps to generate a better source domain performance model, and at the same time alleviates the overfitting problem of the source domain performance model on a small-scale data set.

[0052] Optimizing the hyperparameters of the source domain performance model on the validation set. Specifically, the hyperparameter optimization follows the following steps: selecting a performance score indicator of the source domain performance model, such as mean absolute percentage error (MAPE), mean absolute error (MAE), or coefficient of determination (R2); selecting a set of source domain performance model hyperparameters to be optimized; setting possible values of the source domain performance model hyperparameters; performing a grid search on the value combinations of the source domain performance model hyperparameters, using the average score of 10-fold cross-validation as the score of the source domain performance model, and selecting the optimal hyperparameter combination as the hyperparameters of the source domain performance model.

[0053] Using the test set to evaluate the accuracy of the performance source domain performance model, first selecting a source domain performance model evaluation indicator, using the source domain performance model to obtain the performance prediction output of all samples on the test set, and then calculating the above source domain performance model evaluation indicator according to the true performance value and the predicted performance value, verifying the prediction accuracy of the source domain performance model, and obtaining a trained source domain performance model.

[0054] S105, according to the part of the migration sample of the target cluster, the trained source domain performance model and the migration model, using the support vector regression algorithm, constructing a target cluster performance prediction model.

[0055] Step S105 implements low-cost performance prediction model migration between clusters with similar architecture and different hardware configuration using a support vector regression algorithm.

[0056] Specifically, according to the partial migration samples of the target cluster, the trained source domain performance model and the migration model, a support vector regression algorithm is used to construct a target cluster performance prediction model, including: Step D1: According to the partial migration samples of the target cluster, the migration configuration and the migration label are obtained.

[0057] The partial migration samples are a small number of migration samples, the migration configuration is represented as , and the migration label is represented as .

[0058] Step D2: Using the trained source domain performance model and the migration configuration, the source domain performance prediction value with the same configuration is obtained.

[0059] The trained source domain performance model is represented as , and the source domain performance prediction value with the same configuration is represented as .

[0060] Step D3: The source domain performance prediction value and the migration label form a binary tuple.

[0061] The binary tuple is represented as .

[0062] Step D4: Using a support vector regression algorithm, a migration model is trained between the source cluster and the target cluster to obtain a trained migration model.

[0063] The trained migration model is used to convert the performance prediction model in the source cluster to the performance prediction model of the target cluster.

[0064] The trained migration model is represented as .

[0065] Step D5: The source domain performance model and the trained migration model are combined to obtain the target cluster performance prediction model.

[0066] The trained source domain performance model is combined with the trained migration model in order of sequence to obtain the target cluster performance prediction model .

[0067] S106, input the source cluster and the target cluster into the trained target cluster performance prediction model to realize cross-cluster migration of the performance of the source cluster to the performance of the target cluster by using the trained target cluster performance prediction model, and predict the performance parameters of the cross-cluster migration.

[0068] The distributed machine learning training performance prediction method provided by the application solves the problem that the mapping relationship between parallel configuration and training performance cannot be symbolically modeled for a general architecture distributed machine learning training system, uses a random forest algorithm to construct a black box performance prediction model, and uses a transfer learning technique to solve the cross-cluster migration problem of the performance model at a lower cost. The algorithm framework provided by the method is novel, the prediction accuracy is high, and especially for the scene where the sampling cost of cluster training is high and the training performance data is less, the performance prediction cost can be reduced and the prediction efficiency can be improved. The target cluster performance prediction model of the application can be partially reused between clusters with similar architectures but different hardware configurations.

[0069] To verify the accuracy and effectiveness of the distributed machine learning training performance prediction method of the application, a comparative test is set up, and different workloads are used to experiment on the distributed machine learning performance prediction of the application.

[0070] Considering that accelerating the training process is one of the most common performance optimization goals in the field of distributed machine learning, and that memory overflow and insufficient memory utilization are common factors that limit the training of distributed machine learning systems, the application selects the iteration training time (iter_time) and memory usage (GPU_mem_usage) as the performance indicators to be predicted.

[0071] After feature screening, the parallel configuration parameters are as shown in Table 1, and the encoded parallel configuration parameters are used as input features and as inputs of the performance prediction model of the application. According to the set of core parallel configuration parameters, other commonly used parallel configuration parameters can be calculated, which is beneficial to reduce the number of features, reduce the algorithm complexity and calculation cost, and keep the configuration information relatively complete and independent. In Table 1, ExponentialLR is exponential decay, LinearLR is linear change, and ConstantLR is constant.

[0072] Table 1 Core parallel configuration parameters

[0073] As shown in Table 2, the second generation generative pre-training model (Generative Pre-trained Transformer 2, GPT-2) and the 50-layer residual network (Residual Network 50, ResNet50) are used as the distributed machine learning training workloads, and the news dataset (Australian Broadcasting Corporation News Dataset, ABC News) and the Canadian Institute for Advanced Research 10-class Dataset (CIFAR-10) are used as the training datasets of the workloads.

[0074] Table 2 Experimental workloads

[0075] For two different training performance indicators, the MAPE is used as the evaluation standard for the round training time, and the calculation formula is as follows: The MAE is used as the evaluation standard for the memory occupancy, and the calculation formula is as follows: wherein, is the number of test set samples, the superscript is the true performance value, and the superscript is the performance prediction value, is the true performance value of the i th test set sample, is the performance prediction value of the i th test set sample. The source domain performance model hyperparameter settings of the application are as follows: the number of decision trees n estimator = 50, the support vector machine-based migration model hyperparameter settings are as follows: the kernel function (kernel) is a linear kernel, the number of samples on the source cluster and the target cluster is set to 500 and 100 respectively, wherein the source cluster performance dataset is divided into a training set and a test set in a proportion of 80:20, 50 samples are used for cross-cluster migration on the target cluster, and the remaining 50 samples are used as the test set of the migration model.

[0076] The error evaluation of the source domain performance model and the migrated target cluster performance prediction model for the round training time and the memory occupancy is as shown in Tables 3 and 4.

[0077] Table 3 Prediction error (MAPE) of round training time

[0078] Table 4 Prediction error (MAE) of memory occupancy

[0079] ​Table 4. MAE of GPU memory usage prediction

[0080] Figure 2 Fig. 4 shows a schematic diagram of the round training time model evaluation, Figure 3 Fig. 5 shows a schematic diagram of the GPU memory usage prediction model evaluation, Figure 2 and Figure 3 As shown in Figs. 4 and 5, the results of the source domain performance model prediction accuracy evaluation are shown for the selected round training time and GPU memory usage as two common distributed machine learning training performance evaluation indicators under the GPT-2 and ResNet50 loads, respectively. For the round training time, under the GPT-2 load, the source domain performance model constructed by the random forest algorithm has a clear advantage in accuracy, with an optimal mean absolute percentage error of 9.33%, which is 6.89% lower than the mean absolute percentage error of the Heterogeneous Information Network Performance (HINNPerf) algorithm. Under the ResNet50 load, the source domain performance model constructed by the random forest algorithm has an average absolute error comparable to that of the GBR algorithm, which is only 4.78%. For the GPU memory usage, for the GPT-2 load, the source domain performance model constructed by the random forest algorithm has an average absolute error of only 0.0416, which is 18.27% lower than that of the Deep Performance Analysis Algorithm Performance (DeepPerf) algorithm. For the ResNet50 load, the source domain performance model constructed by the random forest algorithm produces an average absolute error of 0.1684, which is equivalent to that of the Gradient Boosting Regression (GBR) algorithm. The experimental results show that the source domain performance model constructed by the random forest algorithm is significantly better than other comparative algorithms.

[0081] Figure 4 Fig. 6 shows a schematic diagram of the round training time evaluation of the migration model, Figure 5 Fig. 7 shows a schematic diagram of the GPU memory usage time evaluation of the migration model, Figure 4 and Figure 5As shown, the accuracy of the migration model constructed using the support vector regression algorithm is respectively shown. In the migration round training time, the migration model constructed using the support vector regression algorithm has the best performance for the GPT-2 load, and the average absolute percentage error is only 9.43%, which is 6.08% lower than that of the Gaussian process regression (GPR) algorithm, and for the ResNet50 load, the average absolute percentage error is slightly better than that of the LR and BRR algorithms. In the migration memory occupancy performance model, the prediction average absolute error on GPT-2 and ResNet50 is 0.0634 and 0.1112 respectively, which is reduced by 6.90% and 1.42% respectively than the second best algorithm. The migration model constructed using the support vector regression algorithm has good performance in the performance model cross-cluster migration.

[0082] Based on the above experimental data, it can be seen that the distributed machine learning training performance prediction method of the present application has better prediction effect on the round training time and the memory occupancy than other performance prediction algorithms, and the prediction of the round training time is more accurate. The migration algorithm based on support vector regression has lower error than the selected three other migration methods, and the performance on GPT-2 load is better than that on ResNet50 load. The predictability of distributed machine learning performance is higher on GPT-2 network, while there are a large number of skip connections, residual connections and convolution calculations in ResNet50 network, which affect the prediction accuracy and stability of training performance, and the implementation of convolution operation on different software and hardware platforms is not completely the same, which leads to the performance of performance model cross-cluster migration is not as good as GPT-2 load. Overall, the distributed machine learning training performance prediction method proposed in the present application can be applied to the distributed machine learning training performance prediction in general situation.

[0083] Based on the above Figure 1As can be seen from the implementation mode, the embodiment of the present application selects a performance prediction target according to the load characteristics of distributed machine learning training and the performance prediction demand in the target scene; filters all feature parameters according to the performance prediction target to obtain a filtered feature parameter set; collects training performance parameters in the distributed machine learning system by using a random sampling method according to the plurality of filtered feature parameters, and takes the training performance parameters and the corresponding configuration parameters as an original data set; constructs a source domain performance model by using a random forest algorithm according to the configuration parameters and the training performance parameters, and trains the source domain performance model by using part of the original data set to obtain a trained source domain performance model; constructs a target cluster performance prediction model by using a support vector regression algorithm according to part of the migration samples of the target cluster, the trained source domain performance model and the migration model; inputs the source cluster and the target cluster into the trained target cluster performance prediction model to realize the cross-cluster migration of the performance of the source cluster to the performance of the target cluster by using the trained target cluster performance prediction model, and predicts the performance parameters of the cross-cluster migration. In this way, the filtered parallel configuration parameters can reduce the number of features, the machine learning based on the random forest algorithm constructs the source domain performance model, the random forest algorithm is used in the field of distributed machine learning performance prediction, and the target cluster performance prediction model is determined by the transfer learning method to predict the performance parameters of the cross-cluster migration, so that the prediction accuracy is higher and the prediction efficiency is higher; the sampling cost is greatly reduced by the cross-cluster migration method, so that the performance prediction cost is lower.

[0084] Based on the same inventive concept, as an implementation of the above-mentioned distributed machine learning training performance prediction method, the embodiment of the present application further provides a distributed machine learning training performance prediction device. Figure 6 The structure diagram of the distributed machine learning training performance prediction device in the embodiment of the present application is shown in Figure 6 The distributed machine learning training performance prediction device can include: The selection module 601 is configured to select a performance prediction target according to the load characteristics of distributed machine learning training and the performance prediction demand in the target scene. The filtering module 602 is configured to filter all feature parameters according to the performance prediction target to obtain a filtered feature parameter set, and the filtered feature parameter set includes a plurality of filtered feature parameters. The collection module 603 is configured to collect training performance parameters in the distributed machine learning system by using a random sampling method according to the plurality of filtered feature parameters, and take the training performance parameters and the corresponding configuration parameters as an original data set. The first construction module 604 is configured to construct a source domain performance model by using a random forest algorithm according to the configuration parameters and the training performance parameters, and train the source domain performance model by using part of the original data set to obtain a trained source domain performance model. The second construction module 605 is configured to construct a target cluster performance prediction model by using a support vector regression algorithm according to a partial migration sample of the target cluster, the trained source domain performance model and the migration model; The migration module 606 is configured to input the source cluster and the target cluster into the trained target cluster performance prediction model, so as to realize cross-cluster migration of performance from the source cluster to the target cluster by using the trained target cluster performance prediction model, and predict a performance parameter of the cross-cluster migration.

[0085] The screening module 602 is specifically configured to determine initial configuration parameters corresponding to a performance prediction target from all feature parameters of a distributed machine learning training performance file, determine a dependency relationship of features in the initial configuration parameters according to a principle of multi-dimensional parallel training of the distributed machine learning, and decouple initial configuration parameters corresponding to preset dependency relationships in the dependency relationship to obtain a screened feature parameter set.

[0086] The screening module 602 determines a dependency relationship of features in the initial configuration parameters according to a principle of multi-dimensional parallel training of the distributed machine learning, and decouples initial configuration parameters corresponding to preset dependency relationships in the dependency relationship to obtain a screened feature parameter set, including: determining a calculation relationship and a decision relationship of features in the initial configuration parameters according to the principle of multi-dimensional parallel training of the distributed machine learning; taking the initial configuration parameters as vertices, and taking the calculation relationship and the decision relationship as edges to construct a directed graph, the directed graph including a plurality of target connected branches; dividing each target connected branch into a cluster; traversing each cluster and finding vertices with a preset in-degree, and aggregating the vertices with the preset in-degree into the screened feature parameter set, the dependency relationship including the calculation relationship and the decision relationship.

[0087] The collection module 603 is specifically configured to extract one value in a value range corresponding to each of the screened feature parameters according to a random sampling method to generate a configuration file; read and decode each row of features in the configuration file in sequence, and set a value of a corresponding configuration parameter in the distributed machine learning system based on the decoded features to realize training of the distributed machine learning, obtain a training performance parameter, and take the training performance parameter and the configuration parameter as an original data set.

[0088] The first construction module 604 is specifically configured to associate the configuration parameter and the training performance parameter by executing a time stamp, and delete a configuration parameter that is not executed to obtain a vector binary tuple, the vector binary tuple including a configuration parameter vector and a performance index vector; construct a source domain performance model for each component in the performance index vector by using a random forest algorithm; and train the source domain performance model by using a partial original data set to obtain a trained source domain performance model.

[0089] The second construction module 605 is specifically configured to obtain a migration configuration and a migration label according to a partial migration sample of the target cluster; obtain a source domain performance prediction value with the same configuration by using the trained source domain performance model and the migration configuration; compose the source domain performance prediction value and the migration label into a binary tuple; train a migration model between the source cluster and the target cluster by using a support vector regression algorithm to obtain a trained migration model, the trained migration model being used for converting a performance prediction model in the source cluster into a performance prediction model of the target cluster; and combine the source domain performance model and the trained migration model to obtain a target cluster performance prediction model.

[0090] Based on the same inventive concept, the embodiment of the present application further provides an electronic device. Figure 7 The structure diagram of the electronic device in the embodiment of the present application is shown in FIG. 7. Figure 7 As shown in the figure, the electronic device 70 can include at least one processor 701, at least one memory 702 connected with the processor 701, and a bus 703; wherein the processor 701 and the memory 702 complete mutual communication through the bus 703; the processor 701 is used to call the program instruction in the memory 702 to execute the distributed machine learning training performance prediction method in one or more embodiments described above.

[0091] It should be pointed out here that the description of the above distributed machine learning training performance prediction device embodiment is similar to the description of the above distributed machine learning training performance prediction method embodiment, and has the similar beneficial effects as the distributed machine learning training performance prediction method embodiment. For the technical details not disclosed in the embodiment of the distributed machine learning training performance prediction device of the present application, please refer to the description of the distributed machine learning training performance prediction method embodiment of the present application for understanding.

[0092] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for distributed machine learning training performance prediction, the method comprising: The method comprises the following steps: According to the load characteristics of distributed machine learning training and the performance prediction requirements in the target scene, the performance prediction target is selected; According to the performance prediction target, all feature parameters are screened to obtain a set of screened feature parameters, which includes a plurality of screened feature parameters; According to the plurality of screened feature parameters, a random sampling method is used to collect training performance parameters in a distributed machine learning system, and the training performance parameters and corresponding configuration parameters are used as an original data set; According to the configuration parameters and the training performance parameters, a source domain performance model is constructed using a random forest algorithm, and the source domain performance model is trained using part of the original data set to obtain a trained source domain performance model; According to the partial migration samples of the target cluster, the trained source domain performance model and the migration model, a target cluster performance prediction model is constructed using a support vector regression algorithm; The source cluster and the target cluster are input into the trained target cluster performance prediction model to realize the cross-cluster migration of the performance of the source cluster to the performance of the target cluster by using the trained target cluster performance prediction model, and the performance parameters of the cross-cluster migration are predicted.

2. The method of claim 1, wherein, According to the performance prediction target, all feature parameters are screened to obtain a set of screened feature parameters, which includes: In the all feature parameters of the distributed machine learning training performance file, the initial configuration parameters corresponding to the performance prediction target are determined; According to the principle of multi-dimensional parallel training of distributed machine learning, the dependency relationship of the features in the initial configuration parameters is determined, and the initial configuration parameters corresponding to the preset dependency relationship in the dependency relationship are decoupled to obtain the set of screened feature parameters.

3. The method of claim 2, wherein, The dependency relationship includes a calculation relationship and a decision relationship, and according to the principle of multi-dimensional parallel training of distributed machine learning, the dependency relationship of the features in the initial configuration parameters is determined, and the initial configuration parameters corresponding to the preset dependency relationship in the dependency relationship are decoupled to obtain the set of screened feature parameters, which includes: According to the principle of multi-dimensional parallel training of distributed machine learning, the calculation relationship and the decision relationship of the features in the initial configuration parameters are determined; The initial configuration parameters are used as vertices, and the calculation relationship and the decision relationship are used as edges to construct a directed graph, and the directed graph includes a plurality of target connected branches; Each target connected branch is divided into a cluster; Each cluster is traversed and a vertex with a preset in-degree is found, and the vertex with the preset in-degree is aggregated into the set of screened feature parameters.

4. The method of claim 1, wherein, According to the plurality of screened feature parameters, a random sampling method is used to collect training performance parameters in a distributed machine learning system, and the training performance parameters and corresponding configuration parameters are used as an original data set, which includes: According to the random sampling method, a value is extracted in the value domain corresponding to the plurality of screened feature parameters to generate a configuration file; read and decode each line feature in the configuration file in turn, and set the value of the corresponding configuration parameter in the distributed machine learning system based on the decoded feature to realize the training of the distributed machine learning, and obtain a training performance parameter; The training performance parameter and the configuration parameter are used as the original data set.

5. The method of claim 1, wherein, The source domain performance model is constructed by using a random forest algorithm according to the configuration parameter and the training performance parameter, and the source domain performance model is trained by using a part of the original data set, so that a training completed source domain performance model is obtained, which comprises the following steps: By executing the timestamp, the configuration parameter and the training performance parameter are associated, and the configuration parameter that has not been executed is deleted, so that a vector binary tuple is obtained, the vector binary tuple comprises a configuration parameter vector and a performance index vector; The source domain performance model is constructed for each component in the performance index vector by using the random forest algorithm; The source domain performance model is trained by using the part of the original data set, so that the training completed source domain performance model is obtained.

6. The method of claim 1, wherein, The target cluster performance prediction model is constructed by using a support vector regression algorithm according to the part of the migration sample of the target cluster, the training completed source domain performance model and the migration model, which comprises the following steps: According to the part of the migration sample of the target cluster, the migration configuration and the migration label are obtained; The source domain performance prediction value with the same configuration is obtained by using the training completed source domain performance model and the migration configuration; The source domain performance prediction value and the migration label form a binary tuple; The migration model is trained between the source cluster and the target cluster by using the support vector regression algorithm, so that a training completed migration model is obtained, the training completed migration model is used for converting the performance prediction model in the source cluster into the performance prediction model of the target cluster; The source domain performance model and the training completed migration model are combined, so that the target cluster performance prediction model is obtained.

7. The method of claim 1, wherein, The performance prediction target comprises training loss, training accuracy, test loss, test accuracy, round training time, graphics processing unit occupation, video memory occupation and memory occupation.

8. A distributed machine learning training performance prediction apparatus, characterized by, Comprise: The selection module is used for selecting a performance prediction target according to the load characteristics of the distributed machine learning training and the performance prediction demand in the target scene; The screening module is used for screening all feature parameters according to the performance prediction target, so that a screened feature parameter set is obtained, and the screened feature parameter set comprises a plurality of screened feature parameters; The acquisition module is used for acquiring a training performance parameter in a distributed machine learning system by using a random sampling method according to the plurality of screened feature parameters, and using the training performance parameter and the corresponding configuration parameter as an original data set; The first construction module is used for constructing a source domain performance model by using a random forest algorithm according to the configuration parameter and the training performance parameter, and training the source domain performance model by using a part of the original data set, so that a training completed source domain performance model is obtained. The second construction module is configured to construct a target cluster performance prediction model by using a support vector regression algorithm according to a partial migration sample of the target cluster, the trained source domain performance model and the migration model; The migration module is configured to input the source cluster and the target cluster into the trained target cluster performance prediction model, so as to realize cross-cluster migration of the performance of the source cluster to the performance of the target cluster by using the trained target cluster performance prediction model, and predict the performance parameter of the cross-cluster migration.

9. The distributed machine learning training performance prediction apparatus of claim 8, wherein, The screening module is specifically configured to determine the initial configuration parameter corresponding to the performance prediction target from all feature parameters of the distributed machine learning training performance file, determine the dependency relationship of the initial configuration parameter according to the principle of multi-dimensional parallel training of the distributed machine learning, and decouple the initial configuration parameter corresponding to the preset dependency relationship in the dependency relationship to obtain the screened feature parameter set.

10. An electronic device, comprising: The electronic device includes: at least one processor; and at least one memory connected with the processor through a bus; wherein the processor, memory complete mutual communication through the bus; the processor is used to call the program instruction in the memory, to execute the distributed machine learning training performance prediction method in any one of claims 1 to 7.

Images