Construction method of deep learning model parameter optimizer based on two-stage optimization architecture

Abstract

The present invention belongs to the field of deep learning time series prediction, and discloses a method for constructing a deep learning model parameter optimizer based on a two-stage optimization architecture. The method adopts a method for determining the boundary of model optimal parameters by combining a traditional optimizer with a periodic learning rate in step one, and a parameter optimization method for fitting a curve relationship between model training parameters and loss function values ​​in step two. The method for constructing the optimizer not only breaks the limitation of traditional optimizers that only rely on gradient direction updates, but also can stably and effectively improve the prediction accuracy of deep learning models on time series, providing a new research direction for the subsequent improvement of deep learning model prediction performance.

Description

Technical Field

[0001] The present invention relates to the field of deep learning time series prediction, and in particular to a method for constructing a deep learning model parameter optimizer based on a two-stage optimization architecture. Background Art

[0002] Currently, research on deep learning model parameter optimization falls into two main categories: one employs intelligent optimization algorithms to optimize model hyperparameters, such as particle swarm optimization, genetic algorithms, and the sparrow search algorithm; the other employs optimizers to optimize model internal parameters, such as stochastic gradient descent (SGD), mini-batch gradient descent, RMSProp, and Adam. While both approaches aim to unlock the predictive potential of models, the latter, due to its involvement with the internal mechanisms of deep learning models and its lower potential for innovation, is often overlooked in research.

[0003] A deep learning model optimizer is an algorithm used to update the internal parameters of a neural network to reduce the discrepancy between the model's predictions and actual values. Unlike intelligent optimization algorithms like particle swarm optimization and genetic algorithms, which optimize model hyperparameters, an optimizer primarily optimizes internal parameters during model training. It uses various gradient calculation methods to update model parameters during training, aiming to find the optimal parameters for the model in question.

[0004] Currently commonly used deep learning model optimizers such as Adam and SGD that only rely on gradient direction updates are easy and fast to use, but the optimized models have poor stability. They cannot completely and effectively improve the predictive performance of the models, and the optimization performance varies greatly when facing different models. Summary of the Invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a method for constructing a deep learning model parameter optimizer based on a two-stage optimization architecture. The purpose is to break the limitation of traditional optimizers that only rely on gradient direction updates by constructing a new deep learning model parameter optimizer with a two-stage optimization architecture, thereby improving the optimizer's ability to mine model prediction performance and stably and effectively improving the prediction accuracy of the deep learning model.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for constructing a deep learning model parameter optimizer based on a two-stage optimization architecture is provided, comprising the following steps:

[0007] Step 1: Determine the optimal parameter boundaries of the model

[0008] Select the traditional optimizer as the early optimizer to determine the optimal parameter boundaries of the deep learning model, using a fixed learning rate The deep learning model is trained. When the preset number of iterations t is reached, the learning rate is adjusted to a periodic learning rate. The model parameters of the periodic change process of the learning rate are extracted to determine the optimal parameter boundary of the model.

[0009] Step 2: Optimize model parameters

[0010] Fit the better parameters obtained in step 1 to their corresponding model training loss values, and gradually find and obtain the optimal parameters of the deep learning model through the fitted curve.

[0011] Preferably, the traditional optimizer in step one includes Adam optimizer, stochastic gradient descent optimizer, small batch gradient descent optimizer, and RMSProp optimizer; preferably, Adam optimizer.

[0012] Preferably, the periodic learning rate adjustment method in step 1 includes a periodic learning rate of a half-cosine annealing curve, a linear decay learning rate, and a step-like decay learning rate.

[0013] Preferably, in step 1, the periodic learning rate of the half-cosine annealing curve is selected, and the calculation formula is:

[0014] ;

[0015] in, is the cycle length, is the learning rate at time t, is the maximum learning rate, is the minimum learning rate, represents the cosine function, express Divide by The remainder of .

[0016] As a preference, the specific steps for obtaining the optimal parameter boundary of the model by periodic changes in the learning rate are as follows: based on the periodic learning rate formula of the half-cosine annealing curve, set the period length when the learning rate changes periodically , total number of cycles k, number of iterations before the cycle starts t, maximum learning rate and minimum learning rate ; Select the traditional Adam optimizer as the early optimizer, with the maximum learning rate Perform deep learning model parameter training; when the number of training iterations reaches t, start adjusting the learning rate to the periodic learning rate of the half-cosine annealing curve, so that the learning rate periodically increases with the number of iterations during the model training process. and minimum learning rate Fluctuates between, and at the same time extracts the model parameters at the last time point of each cycle during the periodic change of the learning rate and the corresponding loss value at that time point After the model training is completed, k groups of model parameters and their loss values ​​are obtained. The above model parameters surround and form a model better parameter boundary around the optimal parameter with a smaller loss value on the deep learning model loss surface.

[0017] As a preference, the fitting formula in step 2 is:

[0018] ;

[0019] in, is the model training loss value, are model parameters, and is a constant.

[0020] As a preference, the fitting formula in step 2 is:

[0021] ;

[0022] in, is the model training loss value, are model parameters, 、 and c are constants.

[0023] As a preference, the method for optimizing model parameters by fitting the curve relationship between model training parameters and loss function values ​​in step 2 is as follows: Since the model parameters of the deep learning model are gradually optimized during the training process, in order to reduce the impact of poor model parameters on curve fitting, the last two groups of model parameters obtained by the periodic learning rate are selected. and and its loss value and To perform curve fitting; then, on the fitted curve, the early stopping mechanism is introduced to control the optimization process, that is, by gradually reducing the model loss value , to obtain the optimal parameters that can make the model achieve the best prediction effect evaluation index on the training set or validation set , the optimal parameter The model parameters obtained by the final TSO method are used as the prediction model parameters for the remaining test sets.

[0024] Preferably, the prediction effect evaluation index includes the root mean square error of the prediction results of the training set or the validation set, or the sum of the root mean square errors of the two.

[0025] Preferably, the prediction effect evaluation index also includes mean absolute error, mean absolute percentage error or determination coefficient.

[0026] In general, compared with the prior art, the above technical solution conceived by the present invention has the following beneficial effects: 1. Creatively combining the curve function with asymptotic convergence in nature with the loss surface theory of deep learning models to establish a better parameter boundary determination method for the model by combining the traditional gradient descent optimizer with the periodic learning rate in step 1 and the parameter optimization method by fitting the curve relationship between the model training parameters and the loss function value in step 2. or The mathematical model breaks through the limitation of traditional optimizers that only rely on gradient direction updates.

[0027] 2. A two-stage optimization architecture is adopted. In the first stage, a traditional optimizer plus a periodic learning rate is used to explore the optimal model parameter boundaries. In the second stage, parameter fine-tuning is achieved through curve fitting, breaking through the rigidity of the traditional optimizer's single learning rate strategy.

[0028] 3. Compared with traditional optimizers such as stochastic gradient descent (SGD) and Adam, this optimizer performs better on various deep learning models for time series forecasting tasks, and the optimized model has higher prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a diagram of the half cosine annealing learning rate.

[0030] Figure 2 This is a schematic diagram of the loss surface of a deep learning model optimizer based on a two-stage optimization architecture.

[0031] Figure 3 This is a comparison of the prediction performance of the optimizer proposed in this invention compared with other traditional optimizers on multiple deep learning models. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0033] Example 1:

[0034] Taking hourly load data forecasting for a specific province as an example, a long short-term memory (LSTM) network and a gated recurrent unit (GRU) were selected as optimized deep learning models to compare and verify the effectiveness of the present invention. The data used included hourly short-term load data for the province in 2021, hourly average temperature data, and weekday mapping features. Furthermore, the dataset was divided into training, validation, and test sets in a ratio of 8:1:1. For both model training and prediction, one week (168 hours) of historical data was used as the model input, and one hour of future load was used as the output step size.

[0035] Define the basic model hyperparameters: The prediction models of the long short-term memory network (LSTM) and the gated recurrent unit (GRU) after being optimized by the two-stage architecture optimizer are TSO-LSTM and TSO-GRU, respectively. The hyperparameters of the models are as follows:

[0036] TSO-LSTM model: Select Adam as the initial optimizer; select half-cosine annealing learning rate as the learning rate when the learning rate fluctuates periodically; the initial learning rate before the fluctuation is the maximum learning rate Set to 0.008, the minimum learning rate during fluctuation Set it to 0.0006; select the number of iterations t after 380 times and start the periodic change of the learning rate, the cycle length is 10 generations, and the total change is 2 cycles.

[0037] TSO-GRU model: Select Adam as the initial optimizer; select half cosine annealing learning rate as the learning rate when the learning rate fluctuates periodically; the initial learning rate before the fluctuation is the maximum learning rate Set to 0.006, the minimum learning rate during fluctuation Set it to 0.0008; select the number of iterations t after 370 times and start the periodic change of the learning rate, the cycle length is 15 generations, and the total change is 2 cycles.

[0038] Note: The two models are not related during the optimization process and are independent entities. The two models are selected only for comparison to demonstrate the applicability of the optimizer proposed in this invention.

[0039] Step 1: Determine the optimal parameter boundaries of the model

[0040] The traditional Adam optimizer is selected as the early optimizer to determine the optimal parameter boundaries of the deep learning model, using a fixed learning rate The deep learning model is trained. After reaching the preset number of iterations t, the learning rate is adjusted to the periodic learning rate of the half-cosine annealing curve. The model parameters of the periodic change process of the learning rate t are extracted to determine the optimal parameter boundary of the model.

[0041] The periodic learning rate formula of the half-cosine annealing curve is as follows. The period length when the set learning rate changes periodically is , total number of cycles k, number of iterations before the cycle starts t, maximum learning rate and minimum learning rate Fill in the hyperparameters and use the formula:

[0042] ;

[0043] in, is the cycle length, is the learning rate at time t, is the maximum learning rate, is the minimum learning rate, represents the cosine function, express Divide by The remainder of .

[0044] The specific steps to obtain the optimal parameter boundary of the model through the periodic change of the learning rate are as follows: After setting the periodic learning rate formula of the half-cosine annealing curve, select the traditional Adam optimizer as the early optimizer, and use the maximum learning rate Perform deep learning model parameter training; when the number of training iterations reaches t, start adjusting the learning rate to the periodic learning rate of the half-cosine annealing curve, so that the learning rate periodically increases with the number of iterations during the model training process. and minimum learning rate Fluctuates between, and at the same time extracts the model parameters at the last time point of each cycle during the periodic change of the learning rate and the corresponding loss value at that time point After the model training is completed, k groups of model parameters and their loss values ​​are obtained. The above model parameters surround and form a model better parameter boundary around the optimal parameter with a smaller loss value on the deep learning model loss surface.

[0045] Since deep learning model parameters are gradually optimized during training, to reduce the impact of poor model parameters on curve fitting, it is sufficient to change the learning rate by two cycles. This process can obtain two better parameters around the optimal parameter that corresponds to the smallest loss value in the deep neural network model.

[0046] like Figure 1 As shown, the learning rate is adjusted to the periodic learning rate of the half cosine annealing curve, so that the learning rate periodically changes between the maximum learning rate and the maximum learning rate. and minimum learning rate Fluctuations between periods, while extracting the model parameters at the last time point of each period during the periodic change process and , thereby obtaining the optimal parameters corresponding to the smaller loss value in the deep learning model Two better parameters around and and its corresponding loss value and .

[0047] Step 2: Optimize model parameters

[0048] The two optimal parameters obtained in step 1 and The corresponding model training loss value and Fitting is performed according to the following formula, and the optimal parameters of the deep learning model are gradually found and obtained through the fitted curve. The fitting formula is:

[0049] or

[0050] in, is the model training loss value, are model parameters, 、 and c are constants.

[0051] The method of optimizing model parameters by fitting the curve relationship between model training parameters and loss function values ​​is as follows: Since the model parameters of the deep learning model are gradually optimized during the training process, in order to reduce the impact of poor model parameters on curve fitting, the last two groups of model parameters obtained by the periodic learning rate are selected. and and its loss value and To perform curve fitting; then, on the fitted curve, the early stopping mechanism is introduced to control the optimization process, that is, by gradually reducing the model loss value , to obtain the optimal parameters that can make the model achieve the best prediction effect evaluation index on the training set or validation set , the optimal parameter The model parameters obtained by the final TSO method are used as the prediction model parameters for the remaining test sets.

[0052] In this embodiment, the root mean square error (RMSE) of the prediction results of the validation set is used as the evaluation indicator. When the RMSE no longer improves, the parameters corresponding to the loss value are taken as the internal parameters of the model when the model finally predicts the test set.

[0053] This example selects the root mean square error (RMSE), mean absolute error (MAE), mean absolute percentage error (MAPE) and coefficient of determination ( ) was used as the evaluation metric for the model's prediction performance. The Nesterov accelerated gradient (NAG) algorithm and the Adam algorithm, both in the field of model parameter optimization, were used as comparison algorithms. Figure 3 The prediction performance comparison of the optimizer proposed in this paper on LSTM and GRU deep learning models is demonstrated compared with other traditional optimizers.

[0054] In addition to the coefficient of determination ( ) is the closer the value is to 1, the better. The other three indicators, root mean square error (RMSE), mean absolute error (MAE) and mean absolute percentage error (MAPE), are the better. Figure 3 It can be seen that no matter whether LSTM or GRU is selected as the basic model to be optimized, the performance of the optimizer is that TSO is better than Adam, and Adam is better than NAG. It can be seen that the deep learning model parameter optimizer (TSO) based on the two-stage optimization architecture proposed in this invention has certain advantages in improving the model prediction performance compared with traditional optimizers.

[0055] In this example, the traditional Adam optimizer was selected as the pre-optimizer for determining optimal parameter boundaries for deep learning models. In addition to using the Adam optimizer as the pre-optimizer for deep learning model parameter optimization, other optimizers can also be used, such as stochastic gradient descent (SGD), mini-batch gradient descent, and RMSProp. Furthermore, in addition to using the half-cosine annealing curve for cyclical learning rate adjustment, other cyclical learning rate methods can also be used to adjust the learning rate, such as linear decay and step decay.

[0056] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for constructing a deep learning model parameter optimizer based on a two-stage optimization architecture, characterized in that: The following steps are involved: This method is applied to the field of load data forecasting. Step 1: Determine the optimal parameter boundaries of the model An early optimizer that determines the optimal parameter boundaries of deep learning models, using a fixed learning rate α max The deep learning model is trained. When the preset number of iterations t is reached, the learning rate is adjusted to a periodic learning rate. The model parameters of the periodic change process of the learning rate are extracted to determine the optimal parameter boundary of the model. The periodic learning rate adjustment method is selected as the periodic learning rate of the half-cosine annealing curve, and its calculation formula is: Where T is the cycle length, α (t) is the learning rate at time t, α max is the maximum learning rate, α min is the minimum learning rate, cos(·) represents the cosine function, and mod(t,T) represents the remainder of t divided by T; The specific steps to obtain the optimal parameter boundary of the model through the periodic change of the learning rate are as follows: based on the periodic learning rate formula of the half-cosine annealing curve, set the period length T when the learning rate changes periodically, the total number of cycles k, the number of iterations before the cycle starts t, and the maximum learning rate α max and the minimum learning rate α min ; Select Adam optimizer as the early optimizer, with the maximum learning rate α max Conduct deep learning model parameter training; When the number of training iterations reaches t, the learning rate is adjusted to the periodic learning rate of the half-cosine annealing curve, so that the learning rate periodically increases with the number of iterations during the model training process. max and the minimum learning rate α min At the same time, the model parameters W at the last time point of each cycle in the process of periodic changes in the learning rate and the loss value Loss corresponding to that time point are extracted; after the model training is completed, k groups of model parameters and their loss values ​​are obtained. The above model parameters surround and form a model better parameter boundary around the optimal parameter with a smaller loss value on the deep learning model loss surface; Step 2: Optimize model parameters Fit the better parameters obtained in step 1 to their corresponding model training loss values, and gradually find and obtain the optimal parameters of the deep learning model through the fitted curve.

2. The method for constructing a deep learning model parameter optimizer based on a two-stage optimization architecture according to claim 1, characterized in that: The optimizers in step 1 also include stochastic gradient descent optimizer, mini-batch gradient descent optimizer, and RMSProp optimizer.

3. The method for constructing a deep learning model parameter optimizer based on a dual-stage optimization architecture according to claim 1 or 2, characterized in that: The fitting formula in step 2 is: Loss=a*e bW Among them, Loss is the model training loss value, W is the model parameter, and a and b are constants.

4. The method for constructing a deep learning model parameter optimizer based on a dual-stage optimization architecture according to claim 1 or 2, characterized in that: The fitting formula in step 2 is: Loss=a*e bW+c Among them, Loss is the model training loss value, W is the model parameter, and a, b and c are constants.

5. The method for constructing a deep learning model parameter optimizer based on a dual-stage optimization architecture according to claim 3 or 4, characterized in that: In step 2, the method of optimizing model parameters by fitting the curve relationship between model training parameters and loss function values ​​is as follows: the last two sets of model parameters W1 and W2 and their loss values ​​Loss1 and Loss2 obtained by the periodic learning rate are selected for curve fitting; then, on the fitted curve, the optimization process is controlled by introducing an early stopping mechanism, that is, by gradually reducing the model loss value Loss, to obtain the optimal parameter W that can make the model achieve the best prediction effect evaluation index on the training set or validation set. TSO , the optimal parameter W TSO The model parameters obtained by the final TSO method are used as the prediction model parameters for the remaining test sets. TSO is a deep learning model parameter optimizer.

6. The method for constructing a deep learning model parameter optimizer based on a dual-stage optimization architecture according to claim 5, characterized in that: The prediction effect evaluation index includes the root mean square error of the prediction results of the training set or the validation set, or the sum of the root mean square errors of the two.

7. The method for constructing a deep learning model parameter optimizer based on a dual-stage optimization architecture according to claim 6, characterized in that: The prediction effect evaluation index also includes mean absolute error, mean absolute percentage error or determination coefficient.

Images