Gradient descent method based on sequential iterative optimization and application thereof

Abstract

According to the gradient descent method based on sequential iterative optimization and the application of the gradient descent method, the algorithm integrates the advantages of stochastic gradient descent and batch gradient descent, and meanwhile, the algorithm has the characteristics of rapid updating and stable convergence. The core mechanism of the method is that a single sample is selected in sequence to carry out multiple times of iteration parameter updating, multiple optimization parameters are obtained by repeating the process for multiple times, and then the final optimization parameter is determined by adopting the average value of the multiple optimization parameters. According to the method, the robustness of the model is remarkably enhanced through a parameter averaging mechanism, and the method is suitable for parameter optimization in various machine learning tasks, including but not limited to the fields of data fitting, regression analysis, deep learning and the like. Experimental results show that the algorithm shows high prediction precision and good stability in practical applications such as children's urine creatinine content prediction, and has wide application prospects.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning and optimization algorithm, and particularly relates to a gradient descent method based on sequential iterative optimization and application thereof. BACKGROUND

[0002] In machine learning, gradient descent algorithm is a core tool for optimizing model parameters, and is widely used in linear regression, logistic regression and neural network tasks. However, the existing gradient descent method has the following shortcomings: In stochastic gradient descent (SGD), the calculation speed of each iteration is fast, but since only a single sample is used to update the parameters each time, it is easily disturbed by noise, leading to unstable model training. In addition, SGD has a large dependence on initial parameters, which may lead to convergence to a local minimum, and may require more iteration times to converge on large-scale data sets.

[0003] In batch gradient descent (Batch GD), although the gradient is calculated using all samples, the stability of the update direction can be ensured, but the calculation cost is high, especially in the case of large data, the calculation time is long, and it is difficult to run efficiently. In addition, Batch GD needs to load all data at once for gradient calculation, which occupies a large amount of memory, and may be limited when processing large-scale data. At the same time, Batch GD has poor adaptability to dynamic data (such as online learning scenarios), because the gradient of all samples needs to be recalculated for each update.

[0004] In mini-batch gradient descent (Mini-batch GD), although it combines the fast update of SGD and the stability of batch gradient descent, it needs to pre-set the size of the mini-batch, and in some scenarios it may not be able to fully utilize certain statistical characteristics (such as global distribution information) in the data to further optimize model performance.

[0005] To solve the above problems, the present application proposes a gradient descent method based on sequential iterative optimization, which realizes the fast convergence and stability improvement of model training. SUMMARY

[0006] The purpose of the present application is to solve the problems in the prior art and propose a gradient descent method based on sequential iterative optimization and application thereof.

[0007] In order to achieve the above purpose, the present application adopts the following technical scheme: A gradient descent method based on sequential iterative optimization, comprising: Step 1: initialize the regression model parameters , set the learning rate and the iteration threshold iter a, and input the sample x(i); Step 2: sequentially traverse the samples x(i), and each time traverse the current sample x(i) to calculate the updated regression model parameters multiple times iteratively , to obtain the optimized parameter a (i) ; Step 3: repeat Step 2 until all samples are traversed, to obtain multiple optimized parameters a (i) ; Step 4: take the average of the multiple optimized parameters a (i) , to obtain the final optimized parameter .

[0008] Further, the specific steps of Step 2 include: Step 21: set a pointer to sequentially scan each sample x(i); Step 22: after each scan, perform iteratively calculation on the scanned current sample x(i) for a number of iterations, and each iteration updates the regression model parameters according to the parameter update formula , to obtain the optimized parameter a (i) .

[0009] Further, the parameter update formula is:

[0010] wherein, is the regression model parameter, is the learning rate, represents the gradient value of the loss function under the current regression model parameter .

[0011] Further, the specific steps of Step 22 include: Step 221: increase the iteration number by 1; Step 222: calculate the predicted value based on the feature vector X i of the current sample x(i) and the regression model parameter ; Step 223: substitute the predicted value into the error formula to calculate the error value ; Step 224: substitute the error value into the gradient calculation formula to obtain the gradient value ; Step 225: substitute the gradient value into the parameter update formula to update the regression model parameter ; Step 226: Circulating step 221-step 225 until the iteration number is greater than or equal to itera, outputting the final updated regression model parameter, i.e. the optimization parameter a (i) .

[0012] Further, the predicted value The calculation formula is:

[0013] Wherein, The feature vector of sample x(i) is represented, The regression model parameter is represented.

[0014] Further, the error formula is:

[0015] Wherein, The predicted value of sample x(i) is represented, The true value of sample x(i) is represented.

[0016] Further, the gradient calculation formula is: .

[0017] Wherein, The error value of sample x(i) is represented, The feature vector of sample x(i) is represented.

[0018] Further, the final optimization parameter In step 4, the calculation formula is:

[0019] Wherein, m is the sample number, a (i) The optimization parameter is represented.

[0020] The application also provides an application of the gradient descent method based on sequential iteration optimization according to any one of claims 1-8, which is applied to the optimization training of the prediction model, so as to improve the performance of the prediction model.

[0021] Compared with the prior art, the application has the advantages that: 1. The application reduces the noise interference caused by a single sample in the stochastic gradient descent by sequentially traversing the samples, while retaining a faster update speed.

[0022] 2. The application introduces a parameter averaging mechanism, effectively reduces the influence of a single sample on the model, and improves the robustness of the model.

[0023] 3. The gradient descent method based on sequential iteration optimization only uses one sample for updating each time, has low calculation cost, small memory occupation, and can quickly adapt to new data, and is suitable for use in large-scale data sets, limited memory, online learning, and dynamic data updating scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A flowchart of the gradient descent method based on sequential iteration optimization is proposed for the present application.

[0025] Figure 2 A pseudo code of the gradient descent method based on sequential iteration optimization is proposed for the present application.

[0026] Figure 3 A comparison chart of experimental results of the gradient descent method based on sequential iteration optimization in urine creatinine content prediction is proposed for the present application. DETAILED DESCRIPTION

[0027] The present application will be further explained below in conjunction with the accompanying drawings.

[0028] As shown in the drawings, the present application provides a gradient descent method based on sequential iteration optimization, comprising: Figures 1-2 Step 1: initializing regression model parameters , setting learning rate and iteration threshold iter a, and inputting sample x(i). Step 2: sequentially traversing sample x(i) and calculating updated regression model parameters

[0029] a (i) for each traversal of the current sample x(i) multiple times. Step 3: repeating step 2 until all samples are traversed, obtaining multiple optimized parameters a (i) .

[0030] Step 4: taking the average of multiple optimized parameters a (i) to obtain the final optimized parameter

[0031] .

[0032] In step 1, the initial parameter vector of the regression model parameters is set, the learning rate and the iteration threshold iter a are set, the sample x(i) is inputted, the feature vector of the sample x(i) and the true value of the sample x(i) are obtained based on the sample x(i), and finally the parameter list A is initialized as an empty array to store the optimized parameter a (i) . ​

[0033] In step 2, a pointer is first set, and each sample x(i) is scanned in sequence, and after each scan, the current scanned sample x(i) is calculated iteratively for itera times.

[0034] The itera times of iterative calculation of the current scanned sample x(i) specifically includes the following steps: Step 221: incrementing the iteration number by 1; Step 222: based on the feature vector X of the current sample x(i) i and the regression model parameters calculating the predicted value ; Step 223: substituting the predicted value into the error formula to calculate the error value ; Step 224: substituting the error value into the gradient calculation formula to obtain the gradient value ; Step 225: substituting the gradient value into the parameter update formula to update the regression model parameters ; Step 226: repeating steps 221-225 until the iteration number is greater than or equal to itera, and outputting the final updated regression model parameters, i.e., the optimization parameters a (i) .

[0035] In this embodiment, the predicted value is calculated according to the following formula:

[0036] wherein, X represents the feature vector of the sample x(i), and a represents the regression model parameters.

[0037] The error formula is:

[0038] wherein, x represents the predicted value of the sample x(i), and y represents the true value of the sample x(i).

[0039] The gradient calculation formula is: .

[0040] wherein, e represents the error value of the sample x(i), and X represents the feature vector of the sample x(i).

[0041] The parameter update formula is:

[0042] wherein, are regression model parameters, is a learning rate, represents the gradient value of the loss function under the current regression model parameters .

[0043] And before each sample x(i) is iterated for iterations, the iteration number is initialized to 0, so that the iteration number is re-counted for iteration loop, and the optimized parameter a (i) is saved to the parameter list A.

[0044] In the above steps, a single sample x(i) is selected in sequence by sequential traversal, and multiple iteration calculations are performed on the selected single sample x(i) to update the parameters of each sample, avoiding processing all samples at once or relying only on single sample update, reducing the noise interference of single sample, while retaining faster update speed.

[0045] In step 3, since there are multiple samples x(i), step 2 is repeated until all samples are traversed, obtaining multiple optimized parameters a (i) .

[0046] In step 4, the multiple optimized parameters a (i) are averaged to obtain the final optimized parameter to reconstruct the optimized preset model.

[0047] In this embodiment, the calculation formula of the final optimized parameter is:

[0048] wherein, m is the number of samples, and a (i) represents the optimized parameter.

[0049] In the above steps, by averaging the multiple optimized parameters a (i) , the final optimized parameter has higher stability, effectively reducing the influence of single sample on the model and improving the robustness of the model.

[0050] After obtaining the final optimized parameter , the optimized prediction model is based on the final optimized parameter , and the mean square error (MSE) and the goodness of fit (R2) are calculated to evaluate the performance of the optimized prediction model.

[0051] The optimized prediction model is:

[0052] wherein, X represents a matrix composed of feature vectors of all samples, represents final optimization parameters.

[0053] The application also provides an application of the gradient descent method based on sequential iterative optimization according to any one of claims 1-8, which is applied to optimization training of a prediction model, so as to improve the performance of the prediction model.

[0054] The prediction model is applied to data regression analysis, and specifically includes urine creatinine content prediction and house price prediction.

[0055] The following is a specific embodiment taking urine creatinine content prediction as an example: According to Medical Statistics (fourth edition), a local disease research institute investigated the urine creatinine content (mmol / 24 h) of a number of normal children, as shown in Table 1, and Table 1 is as follows:

[0056] According to the data in Table 1, comparative experiments of predicting the urine creatinine content of children are carried out by using the least squares direct solution method (OLS), the stochastic gradient descent method (SGD), the batch gradient descent method (Batch GD), and the gradient descent method based on sequential iterative optimization of the application.

[0057] In the comparative experiment, the least squares method is used as a reference, and the evaluation indexes mean square error (MSE) and goodness of fit (R 2 ) are used to compare the fitting effects of various numerical solution methods, and the comparison of fitting effects of different methods is shown in Table 2 as follows: Figure 2 As shown in Table 2, the experimental results in the comparative experiment of urine creatinine content prediction are shown in Table 2 as follows:

[0058] Figure 3

[0059] In the figure, the horizontal coordinate represents the age, the vertical coordinate represents the urine creatinine content, each point represents the true corresponding relationship between the age and the urine creatinine content of a sample, the fitting straight line represents the relationship between the predicted value and the age, and the horizontal and vertical distance between each point and the fitting straight line represents the difference between the predicted value and the true value.

[0060] In this embodiment, the initial parameter vector is [1.5 0.15], the iteration threshold iterathas a value of 100, and the learning rate is 0.01.

[0061] ​​By comparing the figures, the prediction effects of different methods can be observed intuitively. Ideally, the points should be as close to the fitted straight line as possible, indicating that the predicted values are highly consistent with the true values. As can be seen from the figures: Ordinary Least Squares (OLS): As a reference benchmark for experimental effects comparison.

[0062] Stochastic Gradient Descent (SGD): Since only one sample is used to update the parameters each time, the calculation is small, but the prediction result may be affected by noise, the algorithm is unstable, and often needs to be debugged multiple times and increase the number of iterations.

[0063] Batch Gradient Descent (Batch GD): The algorithm is very stable, the fitting effect is very close to OLS, and a small number of iterations can also achieve good fitting effect. However, all samples participate in calculation each time, resulting in high calculation cost and large memory occupation.

[0064] The sequential iterative optimization gradient descent method of the application: The algorithm is very stable, the fitting effect is very close to OLS, and a small number of iterations can also achieve good fitting effect. The total calculation amount is close to that of Batch Gradient Descent (Batch GD), and the memory occupation is small since only one sample is used to update the parameters each time.

[0065] The experimental results show that the sequential iterative optimization gradient descent method of the application performs well in prediction accuracy and stability, and can quickly converge with a small number of iterations, and has a significant advantage in memory occupation. The method is suitable for various regression analysis tasks, especially in large-scale data sets and dynamic data update scenarios, and has a wide application prospect.

[0066] In addition, compared with Mini-batch Gradient Descent (Mini-batch GD), the sequential iterative optimization method of the application has small calculation amount per iteration, and does not need to set the small batch size in advance, and is suitable for efficient optimization of large-scale data sets.

[0067] From the technical common sense, the application can be realized by other embodiments without departing from the spirit or essential characteristics thereof. Therefore, the above disclosed embodiments are only examples and are not the only ones. All changes within the scope of the application or within the scope equivalent to the application are included in the application.

Claims

1. A gradient descent method based on sequential iterative optimization, characterized in that, include: Step 1: Initialize regression model parameters And set the learning rate And the iteration number threshold itera, then input sample x(i); Step 2: Iterate through the samples x(i) sequentially, and in each iteration, calculate and update the regression model parameters for the current sample x(i) multiple times. The optimized parameter a is obtained. (i) ; Step 3: Repeat step 2 until all samples have been traversed, obtaining multiple optimization parameters a. (i) ; Step 4: Optimize multiple parameters a (i) Take the average value to obtain the final optimized parameters. .

2. The gradient descent method based on sequential iterative optimization according to claim 1, characterized in that: Step 2 specifically includes the following steps: Step 21: Set pointers and scan each sample x(i) sequentially; Step 22: After each scan, perform itera iterations on the current sample x(i) and update the regression model parameters according to the parameter update formula in each iteration. The optimized parameter a is obtained. (i) .

3. The gradient descent method based on sequential iterative optimization according to claim 2, characterized in that: The parameter update formula is: in, These are the parameters of the regression model. For learning rate, This indicates the loss function in the current regression model parameters. The gradient value below.

4. The gradient descent method based on sequential iterative optimization according to claim 3, characterized in that: The specific steps for calculating itera iterations of the current scanned sample x(i) in step 22 include: Step 221: Increment the iteration count by 1; Step 222: Based on the feature vector X of the current sample x(i) i and regression model parameters Calculate the predicted value ; Step 223: Convert the predicted value Substitute the values ​​into the error formula to calculate the error value. ; Step 224: Set the error value Substituting into the gradient calculation formula, we obtain the gradient value. ; Step 225: Adjust the gradient values Substitute the parameters into the parameter update formula to update the regression model parameters. renew; Step 226: Repeat steps 221-225 until the number of iterations is greater than or equal to itera, and output the final updated regression model parameters, i.e., the optimized parameter a. (i) .

5. The gradient descent method based on sequential iterative optimization according to claim 4, characterized in that: The predicted value The calculation formula is: in, Let x(i) be the feature vector of sample x(i). This represents the parameters of the regression model.

6. The gradient descent method based on sequential iterative optimization according to claim 5, characterized in that: The error formula is: in, This represents the predicted value of sample x(i). This represents the true value of sample x(i).

7. The gradient descent method based on sequential iterative optimization according to claim 6, characterized in that: The gradient calculation formula is as follows: in, This represents the error value of sample x(i). Let x(i) represent the feature vector of sample x(i).

8. The gradient descent method based on sequential iterative optimization according to claim 1, characterized in that: The final optimized parameters in step 4 The calculation formula is: Where m is the sample size, a (i) This represents the optimization parameters.

9. An application of the gradient descent method based on sequential iterative optimization as described in any one of claims 1-8, characterized in that: It is applied to the optimized training of prediction models, thereby improving the performance of prediction models.

Images