A method for predicting neural network training time based on deep learning

Abstract

The present invention discloses a method for predicting the training duration of a neural network based on deep learning, which provides predictions for the training duration of deep learning tasks in large-scale clusters to optimize cluster scheduling strategies. The present invention first analyzes the structural characteristics of the neural network and the computational composition of the training process, constructs a computational model for the training duration of a complete neural network with the training duration of a single-layer network as the basic unit, and provides support for using the characteristics of a single-layer model to predict the training duration of neural networks with different structures; secondly, the relationship between model characteristics and training duration is analyzed, and a multi-layer perceptron model is designed to extract the impact of model characteristics on training duration; finally, based on the impact of model characteristics on training duration, a dimensionality reduction rule is formulated to extract key features that have a greater impact on training duration, reduce the dimensionality of the model features, and use the reduced dimensionality model features to train a deep learning model to predict the training duration of neural networks in a distributed environment.

Description

Technical Field

[0001] The present invention relates to a method for predicting the training duration of a neural network based on deep learning, which is used to predict the training duration of a neural network in a large-scale distributed environment. Background Art

[0002] In recent years, with the rapid development of 5G technology, information flow on the internet has reached unprecedented speeds. This rapid information flow has led to a dramatic increase in data volumes, fueling the development of big data and artificial intelligence technologies. In 2016, ResNet technology was introduced for training deep networks, driving the trend toward more complex deep learning models. Due to the dramatic increase in data volumes and the increasing complexity of models, single machines are no longer able to handle the enormous overhead of model training. To address this issue, the industry has begun exploring deep learning technologies in distributed environments.

[0003] Distributed deep learning uses a large number of distributed computing nodes to form a cluster with powerful computing and storage capabilities. This power supports the training of large amounts of data and complex models. Currently, many deep learning frameworks, such as TensorFlow, MXNet, and PaddlePaddle, already support distributed training. However, most distributed deep learning platforms still use cloud / cluster schedulers based on static scheduling strategies, such as Google's Borg cluster scheduler and the Yarn-like schedulers used by Microsoft, Tencent, and Baidu. These schedulers are unable to dynamically adjust resources based on the training time of neural networks within the cluster, resulting in a significant waste of cluster computing resources. Predicting neural network training time in a distributed environment can help design dynamic scheduling strategies for clusters and is a promising solution.

[0004] Initially, time series prediction typically employed machine learning regression algorithms, such as linear regression, random forests, and GBDT. Edelman et al. applied a linear regression model to predict surgical time; Wang et al. used a nearest neighbor-based random forest approach to train multiple regression decision trees to predict bus arrival times; and J. Cheng et al. employed GBDT to predict travel times for different time ranges. These regression methods are not restricted to specific fields or scenarios and possess good versatility. However, they suffer from large error margins in time series prediction and are generally applicable only to scenarios with low sensitivity to temporal fluctuations.

[0005] To reduce the error range of time series forecasting, some researchers have proposed prediction methods that incorporate domain-specific knowledge. By studying the computational characteristics of specific application domains, mathematical models are developed to achieve time series forecasting, such as PALEO and Optimus. PALEO predicts computational time by counting the number of floating-point operations required during a model training period and multiplying the number by a scaling factor. This method assumes that the entire model training process is linearly correlated with the number of floating-point operations and ignores operations that are not linearly correlated with the number of floating-point operations, such as parameter passing. Unlike PALEO, Optimus mathematically summarizes the factors that affect model training, establishes a performance model to evaluate training speed, and predicts model convergence during training based on the number of online resources. Compared to regression methods, these methods reduce the error range of model training time predictions. However, the mathematical analysis model developed for the training process of deep learning tasks is relatively vague and ignores some factors that significantly contribute to the results, resulting in unstable predictions.

[0006] Due to the outstanding performance of deep learning models in various fields, researchers have begun using deep learning-based time series prediction methods to further reduce the errors in time series prediction. Wenquan Xu et al. creatively combined linear regression and deep belief networks to predict time series; PreVIous trained a multi-layer perceptron model to predict the inference time of a convolutional neural network based on the throughput and energy consumption of IoT visual devices; and NC Petersen et al. designed a neural network that combines convolutional and LSTM layers to accurately predict bus arrival times. These methods all achieve high prediction accuracy, but their application to predicting the training time of deep learning models is limited by specific model structures, requiring retraining of their prediction models when encountering new network models.

[0007] To accurately predict the training time of networks with different structures, Daniel Justus et al. divided neural networks into layers and classified them according to computational characteristics (such as convolutional layers and fully connected layers). They then collected model features of single-layer networks to train a multi-layer perceptron model to predict the training time of single-layer models, achieving a high accuracy rate. This method has good versatility. When encountering a new network model, it is only necessary to predict the training time of a single layer based on the hierarchical features of the model. By accumulating the training time of each layer, the training time of the entire model can be predicted. However, the method of Daniel Justus et al. has the following problems when applied in a distributed environment: 1) There is a lack of analysis of the relationship between model features and training time. The large number of collected single-layer network model features include features that have no effect or little effect on training time. 2) A large number of model features need to be collected, and it is usually difficult to obtain all the details of the neural network in a distributed environment. Summary of the Invention

[0008] In response to the shortcomings of the existing technology, the present invention designs and implements a method for predicting the training time of a neural network based on deep learning.

[0009] The present invention provides theoretical support for predicting neural network training time using single-layer model features by constructing a calculation model for the training time of a complete neural network with the training time of a single-layer network as the basic unit, thereby realizing universal prediction of the training time of neural networks with different structures; designs a multi-layer perceptron to extract the impact of model features on training time, realizing effective analysis of the relationship between model features and training time; proposes dimensionality reduction rules to process model feature data, extracts key features that have a greater impact on training time, and reduces the amount of model feature data. Using the feature data after dimensionality reduction processing, a deep learning model is trained to predict neural network training time, improving the feasibility of predicting neural network training time in a distributed environment.

[0010] A method for predicting neural network training time based on deep learning, comprising the following steps:

[0011] Step 1: Construct a computational model for the training time of a complete neural network using the training time of a single-layer network as the basic unit, and analyze the single-layer network model features that affect the training time.

[0012] First, we analyze the computational components of the neural network training process using batch normalization technology and construct a coarse-grained training duration calculation model. A complete neural network training cycle consists of one batch of forward propagation calculations and one backpropagation calculation. During neural network training, multiple iterations of forward and backpropagation calculations are required. Based on the computational components of the neural network training process, we construct a training duration calculation model based on the training time of a complete network.

[0013] Then, based on the hierarchical structure of neural networks, the calculation model for neural network training duration is further refined. A complete neural network is typically composed of multiple layers of neurons, with the output of the current layer serving as the input to the next layer. Forward and backward propagation calculations can be composed of multiple layers of calculations. Combining the above coarse-grained calculation model based on the training duration of the complete network as the basic unit with the hierarchical characteristics of neural networks, a refined calculation model for the training duration of the complete neural network can be obtained, using the training duration of a single layer as the basic unit.

[0014] Finally, we analyze the single-layer network model characteristics that influence training time. Based on the neural network training time calculation model, we achieve the independence of neural network training time prediction and model structure. We analyze the model characteristics of single-layer networks based on the amount of training data and model hyperparameter settings. For neural networks with different structures, we use the calculation model to collect model characteristics of each layer and predict the training time of each layer. By combining the predictions for each layer, we can predict the training time of the entire network.

[0015] Step 2: Design a multi-layer perceptron to extract the impact of model features on the training time of the neural network. For the sake of convenience, the multi-layer perceptron is referred to as a weight model.

[0016] First, we abstract the relationship between model features and training time into a computational representation. For a model feature x, the training time f(x) can be expressed as: f(x) = wx, where w represents the influence of the feature on the training time, i.e., the feature weight.

[0017] Secondly, the feature data is normalized to extract the true feature weights. In the model feature data, each feature has a different range of values, resulting in differences in the size of the feature values. In order to extract the true influence of model features on training time, the present invention uses a normalization algorithm to process the model feature data.

[0018] Finally, based on the computational representation of the relationship between model features and training duration, a weight model is designed to extract feature weights. The input of the weight model is the standardized model feature data, and the output is the predicted single-layer model training duration. The weight model contains a total of 7 hidden layers. The last hidden layer has the same dimension as the input data of the weight model, and its output is dot-multiplied with the input data and passed to the output layer, forming the form f(x) = wx. Therefore, the output of the last hidden layer represents the weight corresponding to the feature.

[0019] Step 3: Propose dimensionality reduction rules, and use the model feature data processed by the dimensionality reduction rules to train a deep learning model to predict the neural network training time. For the sake of convenience, the deep learning model is referred to as a time prediction model.

[0020] First, a weight model is used to extract feature weights from multiple feature data sets to form a weighted data set. Neural network layers can be categorized by computational characteristics, such as convolutional layers and fully connected layers. Different layers correspond to different single-layer model features. Based on the feature data from different single-layer models, a weight model is used to extract the feature weights corresponding to each feature data item to form a weighted data set.

[0021] Next, based on the feature weight dataset, we analyze the overall impact of features on training duration. Different feature values have different impacts on training duration, which can cause fluctuations in feature weights. Based on the feature dataset, we establish the metrics MeanRank and MeanRankStd to measure the overall impact of features.

[0022] Next, based on the overall influence of the features, we design dimensionality reduction rules to reduce the model feature dimensions and data volume. We calculate the MeanRank and MeanRankStd of each feature dataset, and design dimensionality reduction rules to extract the key features that have a greater overall impact on training time. The mathematical representation of the dimensionality reduction rule is as follows:

[0023] Θ={j|MeanRankStd(j)>s∪MeanRank(j)<r,j∈F}

[0024] Use dimensionality reduction rules to process the model feature data set to reduce the dimension of the model feature data.

[0025] Finally, the time prediction model is trained using the reduced feature data. After performing dimensionality reduction on the convolutional and fully connected layer feature data, the convolutional and fully connected layer time prediction models are trained separately. This dimensionality reduction rule reduces the amount of feature data required for the time prediction model while ensuring low error. This improves the feasibility of using deep learning methods to predict neural network training time in a distributed environment.

[0026] The beneficial effects of the present invention are as follows: the present invention can reduce 30% of model features and 25% of training data for the convolutional layer while ensuring high accuracy in predicting the neural network training time; and reduce 20% of model features and 20% of training data for the fully connected layer; thereby reducing the amount of information required to collect for predicting the neural network training time in a distributed environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is an architectural diagram based on deep learning to predict the training time of neural network;

[0028] Figure 2 It is the structure diagram of the weight model. DETAILED DESCRIPTION

[0029] The present invention will be further described below with reference to the accompanying drawings and specific implementation steps.

[0030] A method for predicting neural network training time based on deep learning, comprising the following steps:

[0031] Step 1: Build a computational model for the training time of a complete neural network using the training time of a single-layer network as the unit.

[0032] First, we analyze the computational components of the neural network training process using Batch Normalization technology and construct a coarse-grained training duration calculation model. A complete neural network training session consists of one batch of forward propagation calculations and one backpropagation calculation. During neural network training, multiple iterations of forward and backpropagation calculations are required. Based on the computational components of the neural network training process, we construct a training duration calculation model based on the time required to train a complete network. This model is defined as follows:

[0033]

[0034] Among them, T is the training time of the neural network, E is the number of training epochs, n is the amount of data in the training set, B is the size of the batch, and x is the training time. i is the i-th training data in the current batch, t forward Indicates the time taken for one forward propagation, t backward Indicates the time taken for one back propagation.

[0035] Secondly, based on the hierarchical structure of the network, the calculation model of the neural network training time can be further refined. A complete neural network is usually composed of multiple layers of neurons. The output of the current layer serves as the input of the next layer. The calculation of forward propagation and backpropagation can be composed of multiple layers of calculation. The layered calculation of forward propagation and backpropagation is expressed as follows:

[0036] Forward propagation, data is input into the neural network and output after calculation in each layer. m is the number of neural network layers, x is i is the training data, It represents the time consumption of the forward propagation calculation of the first layer. The hierarchical representation of the forward propagation calculation can be defined as follows:

[0037]

[0038] Back propagation, based on the loss of the output of forward propagation and the expected output, calculates the gradient of the network parameters, from the output layer to the input layer, and adjusts the network parameters layer by layer according to the gradient. Note that m is the number of neural network layers. It represents the time consumption of back propagation calculation of layer l. The hierarchical representation of back propagation calculation can be defined as follows:

[0039]

[0040] Combining the above coarse-grained calculation model with the complete network training time as the basic unit, as shown in formula (1), and the hierarchical calculation representation of the forward and backward propagation of the neural network, as shown in formulas (2) and (3), the neural network training time calculation model can be further refined to use the single-layer training time as the basic unit. The complete network training time calculation model with the single-layer network training time as the basic unit can be defined as follows:

[0041]

[0042] Through the above definition, the present invention models the neural network training duration using the single-layer network computing time as the unit, and theoretically proves that the network training duration can be predicted by the single-layer network characteristics.

[0043] Finally, the single-layer network model features that affect the training time are analyzed, and based on the neural network training time calculation model, the neural network training time prediction and model structure are independent of each other. During the neural network training process, the training set size, model hyperparameter settings and hardware environment will all affect the training time. The factors that affect the neural network training time can be classified according to common features, convolutional layer features, fully connected layer features, recurrent layer features and hardware features. The training time of the convolutional layer and the fully connected layer accounts for the vast majority of the training time of the entire network. Therefore, only the model features of the convolutional layer and the fully connected layer need to be collected. The collected features are shown in the following table according to the above classification criteria:

[0044]

[0045] For neural networks with different structures, according to the computational model, the model features of each layer of the network can be collected to predict the training time of that layer. By combining the single-layer prediction results, the training time of the entire network can be predicted, thus achieving universal prediction for neural networks with different structures.

[0046] Step 2: Design a weight model to extract the impact of model features on neural network training time.

[0047] First, we abstract the relationship between model features and training time into a computational representation. For a model feature x, the training time f(x) can be expressed as: f(x) = wx, where w represents the influence of the feature on the training time, i.e., the feature weight.

[0048] Secondly, the feature data is standardized to extract the real feature weights. In the model feature data, the value range corresponding to each feature is different, which causes the difference in the size of the feature value. In the above-mentioned calculation representation of the relationship between model features and training time, when the original feature data is simply used as x, w will produce deviations due to the difference in values between features, and cannot represent the real influence of the features on the training time. In order to extract the real influence of model features on training time, the present invention uses a standardized algorithm to process the model feature data and maps the feature values to a standard normal distribution.

[0049] Finally, based on the computational representation of the relationship between model features and training time, a weight model is designed to extract feature weights. Figure 2 As shown, the input of the weight model is the standardized model feature data, and the output is the predicted single-layer model training duration. Each layer of neurons is activated by Relu (Reinforced Lu) and then output. The weight model contains 7 hidden layers, with the number of neurons in Layers 1 to 6 being 32, 64, 128, 128, 128, and 128, respectively. The number of neurons in the Weights layer is the same as that in the input layer. The output of the Weights layer is multiplied and summed with the corresponding bits of the input layer data, and then output after Relu activation. The standardized model feature data x is calculated through Layers 1 to 6, and the feature weight w is output. This is then dot-multiplied with the model feature data to form the form f(x) = wx.

[0050] Step 3: Propose dimensionality reduction rules and use the model feature data processed by the dimensionality reduction rules to train the time prediction model.

[0051] First, a weight model is used to extract feature weights from multiple feature data sets to form a weighted data set. Neural network layers can be categorized by computational characteristics, such as convolutional layers and fully connected layers. Different layers correspond to different single-layer model features. Based on the feature data from different single-layer models, a weight model is used to extract the feature weights corresponding to each feature data item to form a weighted data set.

[0052] Secondly, based on the feature weight dataset, we analyze the overall influence of features on training time. In the weight model, for each input feature data x i , the output of the weighted model is: f(x i )=g(x i )x i , where the weight g(x i) is a function that changes with the input data. For different feature data, the corresponding values of each feature are different, and the influence on the training time is also different, which causes the fluctuation of the order of feature influence (i.e., feature weight ranking). Therefore, simply extracting features with larger weights based on the feature weights corresponding to a single feature data will produce a large error and cannot represent the overall influence of the feature on the training time of the neural network. To address this problem, the present invention uses the average ranking MeanRank of feature weights and the average standard deviation MeanRankStd of the rankings to perform an overall analysis of the features. In order to obtain the MeanRank and MeanRankStd of the features, it is first necessary to calculate the feature weight ranking in each feature weight data. The present invention uses the following formula to calculate the weight ranking:

[0053]

[0054] Let Rank(i, j) be the weight ranking of the jth feature in the i-th feature weight data, n be the total number of features, w i，k Indicates the weight of the kth feature in the i-th feature weight data.

[0055] Combined with the feature weight ranking Rank, the average ranking MeanRank of the feature weight and the average standard deviation MeanRankStd of the ranking can be defined as follows:

[0056] The average ranking of feature weights, MeanRank, is based on the model feature datasets and training time data on different hardware devices. The weight model is used to extract the feature weights and the weight ranking is calculated using formula (5) to form the corresponding weight ranking dataset. Let MeanRank(j) be the weighted average ranking of the jth feature, N represents the number of datasets, and M represents the weighted average ranking of the jth feature. d Represents the data volume of the dth weight ranking data set. The average ranking MeanRank of the feature weight can be calculated as follows:

[0057]

[0058] The average standard deviation of the feature weight ranking is MeanRankStd, where MeanRankStd(j) is the average standard deviation of the weight ranking of the jth feature, N is the number of data sets, and M is d represents the data volume of the d-th weight ranking dataset, RankStd(d, j) represents the standard deviation of the j-th feature weight ranking on the d-th test dataset, and the mean ranking MeanRankStd of the feature weight can be calculated as follows:

[0059]

[0060] MeanRank represents the overall distribution of the order of feature influence on different hardware devices, and MeanRankStd represents the overall fluctuation of the order of feature influence. In order to achieve feature dimensionality reduction, this paper develops a unified dimensionality reduction rule based on MeanRank and MeanRankStd to extract the key features that have a greater overall impact on the training time of the neural network. The dimensionality reduction rule can be expressed as the following formula:

[0061] Θ={j|MeanRankStd(j)>s∪MeanRank(j)<r,j∈F} (8)

[0062] Let Θ represent the set of key features selected that have a greater impact on the training time of the neural network, F is the feature set, s and r are constants that need to be set according to the values of MeanRankStd and MeanRank.

[0063] The process of extracting key features using dimensionality reduction rules can be divided into the following two steps:

[0064] 1) Select features whose MeanRankStd is greater than s.

[0065] The stability of the overall feature weight ranking can be intuitively judged from the mean standard deviation (MeanRankStd) of the feature weight ranking. Features with a smaller MeanRankStd have a more stable overall impact on training time, while features with a larger MeanRankStd have a more volatile impact. Excluding features with highly volatile rankings can cause significant deviations in some training time predictions. Therefore, the first step in the dimensionality reduction rule is to extract features with a MeanRankStd greater than s.

[0066] 2) Select features whose MeanRank is less than r.

[0067] After 1) screening, the feature set still contains features with relatively stable rankings (small MeanRankStd). These features include features with a large overall influence on training time (small MeanRank). Therefore, the second step of the dimensionality reduction rule selects features with a MeanRank less than r among the features with stable rankings (small MeanRankStd).

[0068] The process of using dimensionality reduction rules to filter out model features that have a greater impact on neural network training time can be described by the following algorithm:

[0069]

[0070] Finally, the time prediction model is trained using the reduced feature data. To ensure good convergence and short inference time, the time prediction model's structure is identical to the weighted model, except for the dot product between the weights layer and the input layer. The remaining layers are set up identically to the weighted model. After dimensionality reduction of the convolutional and fully connected layer feature data, the convolutional and fully connected layer time prediction models are trained separately. This dimensionality reduction rule reduces the amount of feature data required for the time prediction model while ensuring low error, thereby enabling the use of deep learning methods to predict neural network training duration in a distributed environment.

Claims

1. A method for dynamically adjusting computing resources based on deep learning prediction of neural network training duration, characterized in that: The method comprises the following steps: (1) Combining the computational components of the neural network training process and the hierarchical structural characteristics of the neural network, a computational model for the training time of a complete network is constructed with the training time of a single-layer network as the basic unit; and the model characteristics that affect the training time of a single-layer network are analyzed; The computational components of the training process refer to the forward propagation and backpropagation computational processes of the neural network. The hierarchical structural feature refers to the fact that the neural network structure uses layers as the basic unit, and the computation of the entire network is obtained through the transmission of hierarchical operations. The model features refer to the hyperparameter settings of a single-layer network. (2) Analyze the relationship between model features and training time, and design a multi-layer perceptron model to extract the impact of model features on neural network training time; The multi-layer perceptron model is hereinafter referred to as the weight model, and the influence of the model features on the training time of the neural network is referred to as the feature weight. (3) Using the weight model to extract feature weights, an evaluation index that can measure the overall influence of model features on training time is constructed based on the ranking of feature weights. Based on the evaluation index, a dimensionality reduction rule is designed to achieve dimensionality reduction of model features. The model features after dimensionality reduction are used to train a deep learning model to predict the training time of a single-layer neural network. (4) Dynamically adjust computing resources based on the training duration of the neural network within the cluster; The evaluation indicators include the evaluation of the overall ranking of feature weights and the evaluation of the overall fluctuation degree of the feature weight rankings; for the convenience of naming, the deep learning model used to predict the training time of a single-layer neural network is hereinafter referred to as the time prediction model.

2. The method for dynamically adjusting computing resources based on deep learning prediction of neural network training duration according to claim 1, characterized in that: The calculation model of the complete network training time with the single-layer network training time as the basic unit described in step (1) is established by the following two steps: (1-1) Combined with Batch Normalization technology, one training of a complete neural network consists of a batch of forward propagation calculations and one backpropagation calculation, and this process is iterated multiple times. The training time calculation model with the time consumption of one training of the complete network as the basic unit can be constructed as follows: Among them, T represents the training time of the neural network, E is the number of training epochs, n is the amount of data in the training set, B is the size of the batch, and x is the training time. i is the i-th training data in the current batch, t forward Indicates the time taken for one forward propagation, t backward Indicates the time taken for one back propagation; (1-2) Combined with the hierarchical structure of neural networks, the calculation of forward propagation and backward propagation can be composed of multiple levels of calculation. Further refinement of the calculation model yields a calculation model with the training time of a single-layer network as the basic unit: m is the number of neural network layers, x i is the training data, Indicates the time taken for the forward propagation calculation of the first layer. Indicates the time taken for back propagation calculation of layer l.

3. The method for dynamically adjusting computing resources based on deep learning prediction of neural network training duration according to claim 1, characterized in that: The relationship between the model features and the training time in step (2) can be abstracted as f(x)=wx, where f(x) represents the training time, x is the model feature, and w represents the impact of the feature on the training time, i.e., the feature weight. The input of the weight model described in step (2) is the model feature data after standardization, and the output is the predicted single-layer model training time. Each layer of neurons is output after being activated by ReLu. The number of neurons in the last layer of the hidden layer in the weight model is the same as that in the input layer, and its output is multiplied and added with the corresponding bits of the input layer data, and is output after being activated by ReLu to form the form of f(x)=wx.

4. The method for dynamically adjusting computing resources based on deep learning prediction of neural network training duration according to claim 1, characterized in that: The evaluation indicators for measuring the overall influence of model features on training time in step (3) are the average ranking of feature weights MeanRank and the average standard deviation of rankings MeanRankStd; the dimensionality reduction rule designed based on MeanRank and MeanRankStd can be expressed as: Θ={j|MeanRankStd(j)>s∪MeanRank(j) <r,j∈F} Θ represents the set of key features that have a greater impact on the training time of the neural network, F is the feature set, j represents the jth feature of the feature set, s and r represent constants, which need to be set according to the values of MeanRankStd and MeanRank.

5. The method for dynamically adjusting computing resources based on deep learning prediction of neural network training duration according to claim 1, characterized in that: The structure of the time prediction model described in step (3) only removes the last layer of the hidden layer and the dot product of the input layer compared to the weight model, and the settings of the remaining layers are consistent with the weight model.

Images