Distributed machine learning training performance prediction method based on random forest and transfer learning

Abstract

The invention belongs to the technical field of entrepreneurship guidance, and particularly relates to a distributed machine learning training performance prediction method based on random forest and transfer learning, comprising the following steps: S1, performance prediction target selection; s2, selecting characteristic parameters; s3, collecting a performance sample; s4, training a performance prediction model; and S5, carrying out cross-cluster migration on the performance model. According to the overall structure provided by the embodiment of the invention, aiming at a distributed machine learning training system of a general architecture, the problem that a mapping relation between parallel configuration and training performance cannot be modeled in a symbolic manner is solved, and a black box performance prediction model is constructed by using a random forest algorithm; and a transfer learning technology is adopted to solve the problem of cross-cluster migration of the performance model under the condition of lower cost. The machine learning modeling method based on the random forest algorithm is applied to the field of distributed machine learning performance prediction, has higher performance prediction accuracy compared with other algorithms, and greatly reduces the sampling cost through a transfer learning method.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a distributed machine learning training performance prediction method based on random forest and transfer learning. Background Art

[0002] With the advent of the big data era and the surge of artificial intelligence, the demand for machine learning training tasks is becoming increasingly strong, so distributed machine learning has also become one of the current hot research directions. Due to the explosive growth in the amount of data and the increased complexity of machine learning models, the traditional single-machine single-card training method can no longer meet the resource requirements and time cost requirements of the machine learning training process. Therefore, the application of distributed technology to the machine learning training process, namely distributed machine learning training, has received widespread attention from academia and industry. In order to further shorten the training time of distributed machine learning and optimize the use of computing resources, it is necessary to build a performance prediction model for the mapping relationship between parallel configuration parameters and training performance as a proxy for the actual training performance. In addition, the sampling cost on distributed machine learning training systems is usually expensive. When the cluster hardware configuration changes, in order to quickly build a new performance prediction model on the target cluster with a lower sampling cost, the key technology of cross-cluster migration of distributed machine learning training performance prediction models is required;

[0003] Methods based on analytical modeling, such as the patent application with publication number CN119416866A, entitled “Efficient distributed machine learning training system based on serverless architecture,” disclose an efficient distributed machine learning training system based on serverless architecture, and methods based on learning, such as the patent application with publication number CN202411929662.7, entitled “A method and system for processing device performance statistical data,” disclose a method and system for processing device performance statistical data;

[0004] In the existing technology, the "Efficient Distributed Machine Learning Training System Based on Serverless Architecture" analyzes the communication and aggregation characteristics of serverless architectures and symbolically models the performance prediction model between parameters and training latency, model accuracy, and economic cost, thereby optimizing the configuration parameters, training cost, and performance of the distributed machine learning training system. The shortcomings of this method are that the generation of the performance prediction model is heavily dependent on the analysis of the internal communication and aggregation principles and characteristics of the distributed machine learning training system, requiring a high level of prior knowledge of the system's working principles, and cannot be generalized to distributed machine learning training prediction scenarios under general architectures. Furthermore, it does not consider the low-cost migration of the performance prediction model when the system hardware environment changes.

[0005] "A method and system for processing equipment performance statistics" is applicable to scenarios where historical equipment performance data has time series characteristics and uses discrete working state classification values ​​as prediction targets. This technology has not been used in the field of distributed machine learning training performance prediction, and has not fully selected features for the training configuration parameter characteristics of distributed machine learning in multi-dimensional parallel scenarios. In addition, this method uses long short-term memory networks for equipment performance prediction, which requires frequent forward and back propagation to calculate gradients, involves a large number of matrix operations, and the cost of building the performance prediction model is high. Due to the high sampling cost of distributed machine learning training, the long short-term memory network model also has the risk of overfitting on smaller-scale data sets. Therefore, we propose a distributed machine learning training performance prediction method based on random forests and transfer learning. Summary of the Invention

[0006] This application provides a distributed machine learning training performance prediction method based on random forest and transfer learning to solve the above-mentioned problems.

[0007] This application provides a distributed machine learning training performance prediction method based on random forest and transfer learning, including the following steps:

[0008] S1, performance prediction target selection;

[0009] S2, feature parameter selection;

[0010] S3, performance sample collection;

[0011] S4, performance prediction model training;

[0012] S5. Migrate the performance model across clusters.

[0013] Preferably, the performance prediction target selection includes different indicators for evaluating the performance of training in the distributed machine learning training system, including training loss, training accuracy, test loss, test accuracy, round training time, graphics processing unit (GPU) occupancy, video memory occupancy, and memory occupancy. According to the load characteristics of the distributed machine learning training and the performance prediction requirements in actual scenarios, the performance indicator set p required for performance prediction is selected as follows:

[0014] p={p1, p2, ..., p m}

[0015] Where: m is the total number of performance indicators to be predicted, p1, p2, ..., p m A specific single performance indicator.

[0016] Preferably, the feature parameter selection is intended to screen out a set of configuration parameters that may affect the aforementioned performance prediction target, and use them as input features of the performance prediction model after processing, including redundant feature elimination and dependent feature decoupling.

[0017] Preferably, the redundant feature elimination is used to screen out parallel configuration features that have little impact on the performance of distributed machine learning training, so as to reduce the computational complexity of the algorithm and improve the interpretability of the model;

[0018] The dependent feature decoupling is used to reduce the dependencies between feature parameters, eliminate potential constraints on algorithm input and reduce the number of features.

[0019] Preferably, the performance sample collection is performed by actually executing a distributed machine learning training process in a cluster environment to obtain real performance data of the training as a data set for the performance prediction model, including configuration extraction and performance sampling execution.

[0020] Preferably, the configuration extraction ensures the continuity of the experiment, and a random sampling method is used in advance to extract several configurations to be executed in the feature space to generate representative configurations to reduce the scale of the experiment. The input feature parameter set after feature parameter selection and screening is C = {c1, c2, ..., c n}, the i-th configuration parameter c i The possible value range is D={v il , v i2 ,…,v im}, where v il , v i2 ,…,v im Represents configuration parameter c i The configuration extraction step randomly extracts a value from its value domain for each configuration parameter, namely v i =choice({v il , v i2 ,…,v im}), where choice represents a random value function that randomly extracts an element from a given set. After completing the random extraction of each input feature, all input features v = {v1, v2, ..., v n} as a line of configuration to be executed and written to the configuration file on disk through file operations;

[0021] The performance sampling execution generates a sampling configuration file to be executed based on configuration extraction, reads and decodes each line in the configuration file in turn, sets the values ​​of relevant configuration parameters in the distributed machine learning system, actually executes the distributed machine learning training process, and obtains the actual training performance under each configuration to be sampled.

[0022] Preferably, the performance prediction model training is based on the parallel configuration parameters-training performance data obtained by performance sample collection, and a random forest algorithm is used to construct a mapping relationship between multi-dimensional parallel configuration and training performance to generate a performance prediction model. The specific operations are as follows:

[0023] S41. Data preprocessing: using the execution experiment timestamp to associate the training configuration with the actual training performance, filtering and deleting configuration data for incomplete training processes, and associating the configuration parameters with the performance as a vector binary tuple (c, p), where c represents the configuration parameter vector used as input to the performance prediction model of the present invention, and p represents the performance indicator vector of interest for performance prediction. The vector binary tuple is converted into a data table and serialized into an export file and stored on disk.

[0024] S42, source domain performance model generation, importing data preprocessing data files, reading all configuration parameters and vector tuples (c, p), and performing a performance index vector p for each component p. i , after training, a random forest performance prediction model is generated

[0025]

[0026] in: Representative performance p i The predicted value, f i represents the performance prediction model learned by the random forest algorithm, and c represents the input vector consisting of configuration parameters.

[0027] Preferably, the random forest performance prediction model is obtained by the following steps:

[0028] S421. Randomly divide the obtained distributed machine learning training performance dataset into an initial training set and a test set, where each of the two datasets includes a number of configuration-performance data for calculation;

[0029] S422, using a 10-fold cross-validation method on the initial training set, dividing the initial training set into 10 equal parts, selecting 9 parts each time as training sets to train the random forest model, and 1 part as a validation set to optimize the model hyperparameters;

[0030] S423. Construct a random forest on the training set. The construction of the random forest follows the following steps: generate a root node, randomly extract samples and randomly extract features, and calculate the Gini index for the node according to the following formula:

[0031]

[0032] Perform node splitting and construct a single decision tree, where: p represents the probability distribution of sample classification, K represents the total number of sample classifications, and pk Represents the frequency of the k-th class sample in the node sample set, generates T decision trees and combines them into a random forest, where T is the size of the decision tree, and the average output of all decision trees is used as the output of the random forest;

[0033] S424. Optimize the random forest hyperparameters on the validation set. The hyperparameter optimization follows the following steps: select a scoring indicator for model performance, such as mean absolute percentage error (MAPE), mean absolute error (MAE), or coefficient of determination (R2), select a set of model hyperparameters to be optimized, set possible values ​​for the model hyperparameters, perform a grid search on the value combinations of the model hyperparameters, use the average score of the 10-fold cross-validation as the score of the model, and select the hyperparameter combination with the best score to determine as the hyperparameters of the random forest model.

[0034] Preferably, the model evaluation uses a test set to evaluate the accuracy of the performance prediction model. First, a model evaluation indicator is selected, and the performance prediction output of all samples on the test set is obtained using the performance prediction model. Then, the above-mentioned model evaluation indicator is calculated based on the actual performance value and the predicted performance value to verify the prediction accuracy of the model.

[0035] Preferably, the cross-cluster migration of the performance model uses the support vector regression (SVR) algorithm to achieve low-cost performance prediction model migration between clusters with similar architectures and different hardware configurations, and obtains the migration configuration c based on a small number of migration samples executed on the target cluster. T and migration label p T , use the source domain random forest performance prediction model fs to obtain the source domain performance prediction value ps = fs (c T ), the predicted value of the source domain performance and the actual value of the target domain performance form a binary pair (p s , p T ), train the support vector regression model s(x), and As the target cluster performance prediction model after cross-cluster migration, is the performance prediction value on the target cluster, and c is any given configuration parameter vector of the target cluster.

[0036] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0037] The overall structure provided by the embodiment of the present application, the present invention proposes a distributed machine learning training performance prediction method based on random forest and transfer learning. This method is aimed at the distributed machine learning training system of general architecture, solves the problem that the mapping relationship between parallel configuration and training performance cannot be symbolically modeled, uses the random forest algorithm to build a black box performance prediction model, and adopts transfer learning technology to solve the problem of cross-cluster migration of performance models at a lower cost. The algorithm framework proposed by this method is novel and has high prediction accuracy. Especially for scenarios where cluster training sampling costs are expensive and training performance data is scarce, it can reduce performance prediction costs and improve prediction efficiency. The machine learning modeling method based on the random forest algorithm is used in the field of distributed machine learning performance prediction, and has higher performance prediction accuracy than other algorithms. The sampling cost is greatly reduced through the transfer learning method, and the partial reuse of performance models between clusters with similar architectures but different hardware configurations is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0040] Figure 1 It is a schematic diagram of the overall principle structure of the present invention;

[0041] Figure 2 It is a principle block diagram of characteristic parameter selection of the present invention;

[0042] Figure 3 This is a block diagram of the performance sample collection principle of the present invention;

[0043] Figure 4 This is a block diagram of the performance prediction model training principle of the present invention;

[0044] Figure 5 This is a functional block diagram of the dynamic tracking engine model of the present invention;

[0045] Figure 6 A bar chart showing the evaluation of the round training time model of the present invention;

[0046] Figure 7 A bar chart for evaluating the video memory occupancy prediction model of the present invention;

[0047] Figure 8 A bar chart showing the training time evaluation of the migration model of the present invention;

[0048] Figure 9 A bar chart showing memory occupancy time evaluation of the migration model of the present invention. DETAILED DESCRIPTION

[0049] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] The various embodiments of the present application may be presented in the form of a range. It should be understood that the description in the form of a range is merely for convenience and brevity and should not be construed as a rigid limitation on the scope of the present application. Therefore, it should be considered that the range description has specifically disclosed all possible sub-ranges and single numerical values ​​within the range. For example, it should be considered that the range description from 1 to 6 has specifically disclosed sub-ranges, such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., as well as single numbers within the range, such as 1, 2, 3, 4, 5 and 6, regardless of the range. In addition, whenever a numerical range is indicated in this application, it is intended to include any quoted number (fraction or integer) within the indicated range. Unless otherwise specified, the various raw materials, reagents, instruments and equipment used in this application are all commercially available or can be prepared using existing equipment.

[0051] In this application, unless otherwise specified, the directional words used, such as "upper" and "lower", specifically refer to the directions of the drawings in the accompanying drawings. In addition, in this application, the terms "including", "comprising", etc. mean "including but not limited to". In this application, relational terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. In this application, "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. Wherein A and B can be singular or plural. In this application, "at least one" means one or more, and "plurality" means two or more. "At least one", "at least one of the following" or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, "at least one of a, b, or c" or "at least one of a, b and c" can both mean: a, b, c, ab, i.e. a and b, ac, bc or abc, where a, b, c can be single or multiple.

[0052] like Figures 1-9 As shown: This embodiment of the application provides a distributed machine learning training performance prediction method based on random forest and transfer learning, including the following steps:

[0053] S1, performance prediction target selection;

[0054] S2, feature parameter selection;

[0055] S3, performance sample collection;

[0056] S4, performance prediction model training;

[0057] S5. Migrate the performance model across clusters.

[0058] The performance prediction target selection includes different indicators for evaluating the training performance in the distributed machine learning training system, including training loss, training accuracy, test loss, test accuracy, round training time, graphics processing unit (GPU) occupancy, video memory occupancy, and memory occupancy. According to the load characteristics of the distributed machine learning training and the performance prediction requirements in actual scenarios, the performance indicator set p required for performance prediction is selected as follows:

[0059] p={p1, p2, ..., p m}

[0060] Where: m is the total number of performance indicators to be predicted, p1, p2, ..., p mA specific single performance indicator.

[0061] The feature parameter selection is intended to screen out a set of configuration parameters that may affect the aforementioned performance prediction target, and process them as input features of the performance prediction model, including redundant feature elimination and dependent feature decoupling.

[0062] The redundant feature elimination is used to screen out parallel configuration features that have little impact on the performance of distributed machine learning training, so as to reduce the computational complexity of the algorithm and improve the interpretability of the model;

[0063] The dependent feature decoupling is used to reduce the dependencies between feature parameters, eliminate potential constraints on algorithm input and reduce the number of features.

[0064] The performance sample collection is performed by actually executing the distributed machine learning training process in a cluster environment to obtain the real performance data of the training as the data set of the performance prediction model, including configuration extraction and performance sampling execution.

[0065] The configuration extraction ensures the continuity of the experiment. A random sampling method is used in advance to extract several configurations to be executed in the feature space to generate representative configurations to reduce the scale of the experiment. The input feature parameter set after feature parameter selection and screening is C = {c1, c2, ..., c n}, the i-th configuration parameter c i The possible value range is D={v il , v i2 ,…,v im}, where v il , v i2 ,…,v im Represents configuration parameter c i The configuration extraction step randomly extracts a value from its value domain for each configuration parameter, namely v i =choice({v il , v i2 ,…,v im}), where choice represents a random value function that randomly extracts an element from a given set. After completing the random extraction of each input feature, all input features v = {v1, v2, ..., v n} as a line of configuration to be executed and written to the configuration file on disk through file operations;

[0066] The performance sampling execution generates a sampling configuration file to be executed based on configuration extraction, reads and decodes each line in the configuration file in turn, sets the values ​​of relevant configuration parameters in the distributed machine learning system, actually executes the distributed machine learning training process, and obtains the actual training performance under each configuration to be sampled.

[0067] The performance prediction model training is based on the parallel configuration parameters and training performance data obtained from performance sample collection. The random forest algorithm is used to construct a mapping relationship between multi-dimensional parallel configuration and training performance to generate a performance prediction model. The specific operations are as follows:

[0068] S41. Data preprocessing: using the execution experiment timestamp to associate the training configuration with the actual training performance, filtering and deleting configuration data for incomplete training processes, and associating the configuration parameters with the performance as a vector binary tuple (c, p), where c represents the configuration parameter vector used as input to the performance prediction model of the present invention, and p represents the performance indicator vector of interest for performance prediction. The vector binary tuple is converted into a data table and serialized into an export file and stored on disk.

[0069] S42, source domain performance model generation, importing data preprocessing data files, reading all configuration parameters and vector tuples (c, p), and performing a performance index vector p for each component p. i , after training, a random forest performance prediction model is generated

[0070]

[0071] in: Representative performance p i The predicted value, f i represents the performance prediction model learned by the random forest algorithm, and c represents the input vector consisting of configuration parameters.

[0072] The random forest performance prediction model is obtained by the following steps:

[0073] S421. Randomly divide the obtained distributed machine learning training performance dataset into an initial training set and a test set, where each of the two datasets includes a number of configuration-performance data for calculation;

[0074] S422, using a 10-fold cross-validation method on the initial training set, dividing the initial training set into 10 equal parts, selecting 9 parts each time as training sets to train the random forest model, and 1 part as a validation set to optimize the model hyperparameters;

[0075] S423. Construct a random forest on the training set. The construction of the random forest follows the following steps: generate a root node, randomly extract samples and randomly extract features, and calculate the Gini index for the node according to the following formula:

[0076]

[0077] Perform node splitting and construct a single decision tree, where: p represents the probability distribution of sample classification, K represents the total number of sample classifications, and pk Represents the frequency of the k-th class sample in the node sample set, generates T decision trees and combines them into a random forest, where T is the size of the decision tree, and the average output of all decision trees is used as the output of the random forest;

[0078] S424. Optimize the random forest hyperparameters on the validation set. The hyperparameter optimization follows the following steps: select a scoring indicator for model performance, such as mean absolute percentage error (MAPE), mean absolute error (MAE), or coefficient of determination (R2), select a set of model hyperparameters to be optimized, set possible values ​​for the model hyperparameters, perform a grid search on the value combinations of the model hyperparameters, use the average score of the 10-fold cross-validation as the score of the model, and select the hyperparameter combination with the best score to determine as the hyperparameters of the random forest model.

[0079] The model evaluation uses a test set to evaluate the accuracy of the performance prediction model. First, a model evaluation indicator is selected, and the performance prediction output of all samples on the test set is obtained using the performance prediction model. Then, the above model evaluation indicator is calculated based on the actual performance value and the predicted performance value to verify the prediction accuracy of the model.

[0080] The cross-cluster migration of the performance model uses the support vector regression (SVR) algorithm to achieve low-cost performance prediction model migration between clusters with similar architectures and different hardware configurations. The migration configuration c is obtained based on a small number of migration samples executed on the target cluster. T and migration label p T , use the source domain random forest performance prediction model fs to obtain the source domain performance prediction value ps = fs (c T ), the predicted value of the source domain performance and the actual value of the target domain performance form a binary pair (p s , p T ), train the support vector regression model s(x), and As the target cluster performance prediction model after cross-cluster migration, is the performance prediction value on the target cluster, and c is any given configuration parameter vector of the target cluster.

[0081] Example

[0082] Considering that accelerating the training process is one of the most common performance optimization goals in the field of distributed machine learning, and that graphics memory overflow and insufficient graphics memory utilization are common factors that limit the training of distributed machine learning systems, this embodiment selects round training time (iter_time) and graphics memory usage (GPU_mem_usage) as the performance indicators to be predicted.

[0083] After feature screening, the parallel configuration parameters for this embodiment are shown in Table 1. These parallel configuration parameters are encoded and used as input features for the performance prediction model of the present invention. Based on this set of core parallel configuration parameters, other commonly used parallel configuration parameters can be calculated, which helps reduce the number of features, algorithm complexity, and computational cost, while maintaining the relative integrity and independence of configuration information.

[0084] Table 1 Core parallel configuration parameters

[0085]

[0086] This embodiment uses the second-generation generative pre-training model (GPT-2) and the 50-layer residual network (ResNet50) as the workloads for distributed machine learning training, and uses the American Broadcasting Corporation News dataset (ABCNews) and the Canadian Institute for Advanced Research-10 (CIFAR-10) dataset as the training datasets for the workloads, respectively. The detailed information is listed in Table 2.

[0087] Table 2 Experimental workload

[0088]

[0089] For two different training performance indicators, the round training time uses the mean absolute percentage error (MAPE) as the evaluation criterion, which is calculated as follows:

[0090]

[0091] Where n is the number of test set samples, the superscript true is the actual performance value, and the superscript pred is the performance prediction value. The mean absolute error (MAE) is used as the evaluation criterion for video memory usage, and its calculation formula is:

[0092]

[0093] The symbols in the formula have the same meaning as above.

[0094] The hyperparameters for the random forest performance prediction model described in this example are set to n_estimator = 50 decision trees and a linear kernel for the support vector machine migration model. The number of samples on the source and target clusters is set to 500 and 100, respectively. The source cluster performance dataset is split into a training set and a test set in an 80%:20% ratio. Fifty samples are used on the target cluster for cross-cluster migration, and the remaining 50 samples serve as the test set for the migration model.

[0095] The error evaluation of the source domain performance model and the target domain performance model after migration in terms of round training time and video memory usage is shown in Tables 3 and 4.

[0096] Table 3. Round training time prediction error (MAPE)

[0097]

[0098] Table 4 Memory occupancy prediction error (MAE)

[0099]

[0100] From Tables 3 and 4 and Figure 6-9 It can be seen from the data that the performance prediction method based on random forest is better than other performance prediction algorithms to varying degrees in terms of the prediction effect of round training time and video memory occupancy, and the prediction of round training time is more accurate. The migration algorithm based on support vector regression has a lower error than the three other selected migration methods, and its performance on the GPT-2 load is better than that on the ResNet50 load. The predictability of distributed machine learning performance on networks such as GPT-2 is relatively high, while the ResNet50 network contains a large number of jump connections, residual connections, convolution calculations and other links. These links have an impact on the prediction accuracy and stability of training performance, and the implementation of convolution operations on different software and hardware platforms is not exactly the same, resulting in the performance of cross-cluster migration of performance models being inferior to the GPT-2 load. In general, the method proposed in the present invention can be applied to the prediction of distributed machine learning training performance under general circumstances.

[0101] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but rather is intended to conform to the broadest scope consistent with the principles and novel features claimed herein.

Claims

1. A distributed machine learning training performance prediction method based on random forest and transfer learning, characterized in that: The steps include: S1, performance prediction target selection; S2, feature parameter selection; S3, performance sample collection; S4, performance prediction model training; S5. Migrate the performance model across clusters.

2. The method for predicting distributed machine learning training performance based on random forest and transfer learning according to claim 1, characterized in that: The performance prediction target selection includes different indicators for evaluating the training performance in the distributed machine learning training system, including training loss, training accuracy, test loss, test accuracy, round training time, graphics processing unit (GPU) occupancy, video memory occupancy, and memory occupancy. According to the load characteristics of the distributed machine learning training and the performance prediction requirements in actual scenarios, the performance indicator set p required for performance prediction is selected as follows: p={p1,p2,…,p m } Where: m is the total number of performance indicators to be predicted, p1, p2, ..., p m A specific single performance indicator.

3. The method for predicting distributed machine learning training performance based on random forest and transfer learning according to claim 1, characterized in that: The feature parameter selection is intended to screen out a set of configuration parameters that may affect the aforementioned performance prediction target, and process them as input features of the performance prediction model, including redundant feature elimination and dependent feature decoupling.

4. The method for predicting distributed machine learning training performance based on random forest and transfer learning according to claim 3, characterized in that: The redundant feature elimination is used to screen out parallel configuration features that have little impact on the performance of distributed machine learning training, so as to reduce the computational complexity of the algorithm and improve the interpretability of the model; The dependent feature decoupling is used to reduce the dependencies between feature parameters, eliminate potential constraints on algorithm input and reduce the number of features.

5. The method for predicting distributed machine learning training performance based on random forest and transfer learning according to claim 1, characterized in that: The performance sample collection is performed by actually executing the distributed machine learning training process in a cluster environment to obtain the real performance data of the training as the data set of the performance prediction model, including configuration extraction and performance sampling execution.

6. The method for predicting distributed machine learning training performance based on random forest and transfer learning according to claim 5, characterized in that: The configuration extraction ensures the continuity of the experiment. A random sampling method is used in advance to extract several configurations to be executed in the feature space to generate representative configurations to reduce the scale of the experiment. The input feature parameter set after feature parameter selection is C = {c1, c2, ..., c n }, the i-th configuration parameter c i The possible value range is D={v il , v i2 ,…,v im }, where v il , v i2 ,…,v im Represents configuration parameter c i The configuration extraction step randomly extracts a value from its value domain for each configuration parameter, namely v i =choice({v il , v i2 ,…,v im }), where choice represents a random value function that randomly extracts an element from a given set. After completing the random extraction of each input feature, all input features v = {v1, v2, ..., v n } as a line of configuration to be executed and written to the configuration file on disk through file operations; The performance sampling execution generates a sampling configuration file to be executed based on configuration extraction, reads and decodes each line in the configuration file in turn, sets the values ​​of relevant configuration parameters in the distributed machine learning system, actually executes the distributed machine learning training process, and obtains the actual training performance under each configuration to be sampled.

7. The method for predicting distributed machine learning training performance based on random forest and transfer learning according to claim 1, characterized in that: The performance prediction model training is based on the parallel configuration parameters and training performance data obtained from performance sample collection. The random forest algorithm is used to construct a mapping relationship between multi-dimensional parallel configuration and training performance to generate a performance prediction model. The specific operations are as follows: S41. Data preprocessing: using the execution experiment timestamp to associate the training configuration with the actual training performance, filtering and deleting configuration data for incomplete training processes, and associating the configuration parameters with the performance as a vector binary tuple (c, p), where c represents the configuration parameter vector used as input to the performance prediction model of the present invention, and p represents the performance indicator vector of interest for performance prediction. The vector binary tuple is converted into a data table and serialized into an export file and stored on disk. S42, source domain performance model generation, importing data preprocessing data files, reading all configuration parameters and vector tuples (c, p), and performing a performance index vector p for each component p. i , after training, a random forest performance prediction model is generated in: Representative performance p i The predicted value, f i represents the performance prediction model learned by the random forest algorithm, and c represents the input vector consisting of configuration parameters.

8. The method for predicting distributed machine learning training performance based on random forest and transfer learning according to claim 7, characterized in that: The random forest performance prediction model is obtained by the following steps: S421. Randomly divide the obtained distributed machine learning training performance dataset into an initial training set and a test set, where each of the two datasets includes a number of configuration-performance data for calculation; S422. Use the 10-fold cross-validation method on the above initial training set to divide the initial training set into 10 parts on average. Select 9 parts each time as training sets to train the random forest model, and select 1 part as the validation set to optimize the model hyperparameters. S423. Construct a random forest on the training set. The construction of the random forest follows the following steps: generate a root node, randomly extract samples and randomly extract features, and calculate the Gini index for the node according to the following formula: Perform node splitting and construct a single decision tree, where: p represents the probability distribution of sample classification, K represents the total number of sample classifications, and p k Represents the frequency of the k-th class sample in the node sample set, generates T decision trees and combines them into a random forest, where T is the size of the decision tree, and the average output of all decision trees is used as the output of the random forest; S424. Optimize the random forest hyperparameters on the validation set. The hyperparameter optimization follows the following steps: select a scoring indicator for model performance, such as mean absolute percentage error (MAPE), mean absolute error (MAE), or coefficient of determination (R2), select a set of model hyperparameters to be optimized, set possible values ​​for the model hyperparameters, perform a grid search on the value combinations of the model hyperparameters, use the average score of the 10-fold cross-validation as the score of the model, and select the hyperparameter combination with the best score to determine as the hyperparameters of the random forest model.

9. The method for predicting distributed machine learning training performance based on random forest and transfer learning according to claim 1, characterized in that: The model evaluation uses a test set to evaluate the accuracy of the performance prediction model. First, a model evaluation indicator is selected, and the performance prediction output of all samples on the test set is obtained using the performance prediction model. Then, the above model evaluation indicator is calculated based on the actual performance value and the predicted performance value to verify the prediction accuracy of the model.

10. The method for predicting distributed machine learning training performance based on random forest and transfer learning according to claim 1, characterized in that: The cross-cluster migration of the performance model uses the support vector regression (SVR) algorithm to achieve low-cost performance prediction model migration between clusters with similar architectures and different hardware configurations. The migration configuration c is obtained based on a small number of migration samples executed on the target cluster. T and migration label p T , use the source domain random forest performance prediction model fs to obtain the source domain performance prediction value ps = fs (c T ), the predicted value of the source domain performance and the actual value of the target domain performance form a binary pair (p s , p T ), train the support vector regression model s(x), and As the target cluster performance prediction model after cross-cluster migration, is the performance prediction value on the target cluster, and c is any given configuration parameter vector of the target cluster.

Images