Three-item minimization subspace conjugate gradient optimization method

CN119003961APending Publication Date: 2024-11-22ZHEJIANG GONGSHANG UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411019661.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

[0003]然而,在三项共轭梯度法中,gk+1,sk,yk等向量前的系数都是固定的,这些固定的系数常常不能很好地适应优化问题复杂度的上升和求解规模的扩大所带来的挑战

Benefits of technology

[0042] 1. This invention chooses a new vector combination that is different from the classic three-term conjugate gradient method: s k , y k , y k-1 , the subspace Ω formed by combining the vectors k+1 Solve the sub-optimization problem to determine the new search direction, and the search direction satisfies the sufficient descent condition and the Dai-Liao conjugation condition, so that the present invention reduces the number of evaluations of the objective function during the optimization process, thereby reducing the cost of practical applications It provides a more feasible and practical solution to solve complex optimization problems with less computational cost. At the same time, under reasonable assumptions, it is proved that the new search direction has global convergence and reaches convergence with fewer iterations. Its fast convergence and high efficiency make this algorithm have great potential in solving large-scale optimization problems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003961A_ABST
    Figure CN119003961A_ABST
Patent Text Reader

Abstract

The invention discloses a three-item minimization subspace conjugate gradient optimization method, which specifically comprises the following steps of: taking an initial iteration point, setting an initial iteration frequency as 0, calculating an initial gradient of a function, determining an initial search direction, judging whether a Himmblau termination condition is met or whether the iteration frequency reaches a set value or not, if so, returning a decision variable, and if not, executing the step of executing the step of executing the step of executing the step of executing the step of executing the step of executing the step of executing the step of executing the step of executing the step of executing the step of executing the step. And if not, determining a search step length based on a Wolfe line search criterion, judging whether an Andrii acceleration strategy is implemented, a decision variable is updated, a negative gradient restart strategy is executed, 1 is accumulated on the number of iterations, and returning to judge whether a Himmblau termination condition is met or the number of iterations reaches a set value. The invention is based on sk. According to the method, a new search direction meeting a full descent condition and a Dai-Liao conjugation condition is constructed in a subspace formed by a three-item minimization subspace conjugate gradient method, a new search direction meeting the full descent condition and the Dai-Liao conjugation condition is constructed in a subspace formed by a three-item minimization subspace conjugate gradient method, and the convergence speed is higher and errors are lower than those of an existing three-item minimization subspace conjugate gradient method in solving the unconstrained optimization problem.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of unconstrained optimization algorithms, and specifically relates to a three-term minimization subspace conjugate gradient optimization method. k ,y k ,y k-1 The spanned subspace Ω k+1 A new search direction that satisfies the sufficient descent condition and the Dai-Liao conjugation condition is constructed, achieving faster convergence speed and lower error. Background Art

[0002] The conjugate gradient method (CG) is one of the most commonly used optimization algorithms with a fast convergence speed in applications. So far, many CG methods have been developed to improve computational performance and are widely used to solve real-world problems. However, the general conjugate gradient method often frequently uses negative gradients as the restart direction, which greatly reduces the efficiency of the conjugate gradient method. To overcome this shortcoming, Beale first proposed the three-term conjugate gradient method. Compared with the traditional conjugate gradient method, the three-term conjugate gradient method can converge to the optimal solution faster, effectively improving the efficiency of the optimization algorithm.

[0003] However, in the three-term conjugate gradient method, g k+1 ,s k ,y k The coefficients before the vectors are fixed, and these fixed coefficients often cannot adapt well to the challenges brought by the increase in the complexity of the optimization problem and the expansion of the solution scale. In order to meet these challenges, some scholars use the idea of ​​the trust region method to solve the problem based on g. k+1 ,s k ,y k The search direction is determined by solving sub-optimization problems in the subspace spanned by equal vectors, which leads to the development of SMCG algorithm, STT algorithm and TTS algorithm. The numerical results show that they are better than some classic conjugate gradient methods.

[0004] The three-term minimization subspace conjugate gradient method often shows faster convergence speed and stronger robustness in solving large-scale unconstrained optimization problems. Therefore, more and more scholars have made meaningful explorations on the minimization subspace conjugate gradient method. Summary of the invention

[0005] Since the three-term minimization subspace conjugate gradient method often shows faster convergence speed and stronger robustness, the present invention proposes a three-term minimization subspace conjugate gradient optimization method (denoted as OURS), in which s k ,y k ,y k-1 The spanned subspace Ω k+1A new search direction that satisfies the sufficient descent condition and the Dai-Liao conjugation condition is constructed, which achieves faster convergence speed and lower error than the existing three-term minimization subspace conjugate gradient method in solving unconstrained optimization problems.

[0006] The present invention provides a three-term minimization subspace conjugate gradient optimization method, comprising the following steps:

[0007] S1: Take the initial iteration point x0, the initial iteration number is recorded as 0, and calculate the initial gradient of the function f(x) And determine the initial search direction d0 = -g0;

[0008] S2: If the function value or gradient of the function f(x) satisfies the Himmeblau termination condition, or the number of iterations reaches epoch, the iteration terminates and the decision variable is returned, otherwise the next step is executed;

[0009] S3: Based on Wolf e The line search criterion determines the search step size;

[0010] S4: Determine whether to implement the Andrei acceleration strategy based on the search direction and the search step size in step S3, and update the decision variables;

[0011] S5: Calculate the dimension of the three-term subspace;

[0012] S6: Based on the dimensions of the three-term subspace in step S5, calculate the search direction for the next iteration;

[0013] S7: execute the negative gradient restart strategy;

[0014] S8: The number of iterations is increased by 1, and the process returns to step S2.

[0015] Furthermore, the step S2 includes the following sub-steps:

[0016] S21: Order Among them, x k is the iteration point of step k;

[0017] S22: Record the gradient in the k-th iteration If Δ<∈1 or ||g k || ∞ ≤∈3 or the number of iterations reaches epoch, the iteration terminates and the decision variable (i.e. x k ), otherwise, execute the next step. Among them, ∈1, ∈2, ∈3 are the three parameters of Himmeblau termination condition.

[0018] Furthermore, step S3 specifically calculates the search step length α based on the gradient and search direction in step S1. k To meet Among them, s k =x k+1 -x k , d k is the search direction of the kth step, ρ and σ are two parameters of the Wolfe line search criterion, and 0<ρ≤σ<1.

[0019] Furthermore, the step S4 includes the following sub-steps:

[0020] S41: Calculation where y k =g k+1 -g k ;

[0021] S42: If Then implement the Andrei acceleration strategy and calculate the acceleration parameters And update the decision variables, that is, let x k+1 =x k +ζ k α k d k , otherwise x k+1 =x k +α k d k .

[0022] Furthermore, the three subspaces in step S5 are composed of s k =x k+1 -x k ,y k =g k+1 -g k and k-1 =g k -g k-1 Spanning the linear subspace Ω k+1 , and the dimension of the subspace may have three cases, namely dimΩ k+1 =3, 2 or 1.

[0023] Furthermore, the step S6 includes the following sub-steps:

[0024] S61: Assumption d k+1 =μ k s k +v k y k +τ k y k-1 , where the coefficient To be determined;

[0025] S62: D k+1 =μ k s k +v ky k +τ k y k-1 , substitute into the minimization problem , solve for μ k , v k , τ k ; Among them B k+1 is the function f(x) at x k+1 The approximation of the Hessian matrix, assuming that B k+1 is symmetric and positive and satisfies the quasi-Newton condition, that is, B k+1 s k =y k .

[0026] S63: If dimΩ k+1 =1, then d k+1 =-g k+1 ;

[0027] If dimΩ k+1 =2 and s k With y k Linearly independent, then d k+1 =μ k s k +v k y k ,in

[0028]

[0029] If dimΩ k+1 =2 and s k With y k-1 Linearly independent, then d k+1 =μ k s k +τ k y k-1 ,in

[0030]

[0031] If dimΩ k+1 =3, then d k+1 =μ k s k +v k y k +τ k y k-1 ,in

[0032]

[0033] Furthermore, in step S63, if dimΩ k+1 =2 and s k With yk Linearly independent, to ensure A k Positive, take Therefore

[0034]

[0035] If dimΩ k+1 =2 and s k With y k-1 Linear independence, to ensure Positive, take Therefore

[0036]

[0037] If dimΩ k+1 =3, to ensure Positive, take in ξ is a constant greater than 1, so

[0038]

[0039]

[0040] Furthermore, the negative gradient restart strategy in step S7 is: if Then d k+1 Reset to -g k+1 Then execute step S8, otherwise execute step S8 directly.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. This invention selects a new vector combination different from the classic three-term conjugate gradient method: s k ,y k ,y k-1 , by combining the vectors in the subspace Ω k+1 The sub-optimization problem is solved to determine the new search direction, and the search direction satisfies the sufficient descent condition and the Dai-Liao conjugate condition, so that the present invention reduces the number of evaluations of the objective function during the optimization process, thereby reducing the computational cost in practical applications and providing a more feasible and practical solution for solving complex optimization problems. At the same time, under reasonable assumptions, it is proved that the new search direction has global convergence and reaches convergence with fewer iterations. Its fast convergence and high efficiency make the algorithm have great potential in solving large-scale optimization problems.

[0043] 2. In the process of solving the sub-optimization problem and determining the search direction, the present invention uses the approximate Hessian matrix Bk+1 With vector s k ,y k ,y k-1 All quadratic terms are estimated to ensure that the sub-optimization problem has a unique solution, and a simpler and more accurate parameter expression is obtained, so that the present invention has higher efficiency and speed in the optimization problem, and is more robust in the process of performance optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The present invention is a flowchart of a three-term minimization subspace conjugate gradient optimization method.

[0045] Figure 2 This is a performance comparison curve of the present invention and the other two algorithms regarding the number of iterations.

[0046] Figure 3 This is a performance comparison curve of the present invention and the other two algorithms regarding the number of function value calculations.

[0047] Figure 4 This is a performance comparison curve of the present invention and the other two algorithms regarding CPU running time. DETAILED DESCRIPTION

[0048] The following is a further detailed description of the embodiments of the present invention in conjunction with the accompanying drawings and examples. The detailed description of the following embodiments and the accompanying drawings are used to exemplarily illustrate the principles of the present invention, but cannot be used to limit the scope of the present invention. Those skilled in the art can combine the technical features in the embodiments to form new embodiments without creative work, and these new embodiments are also covered within the scope of protection of the present invention.

[0049] The present invention is described in detail below by taking minimizing the two-dimensional Rosenbrock function as an example and in conjunction with the accompanying drawings. The two-dimensional Rosenbrock function is shown in formula (1):

[0050]

[0051] Its global minimum point and minimum value are: x * =(1,1),f(x * )=0.

[0052] In order to avoid confusion, in the following specific implementation steps, unless otherwise specified, the subscripts of the variables represent the number of iterations rather than the components of the vector. Vectors k =x k+1 -x k =α k d k , vector y k =gk+1 -g k , where α k is the search step length of the k-th iteration, d k is the search direction for the k-th iteration.

[0053] like Figure 1 As shown, the present invention provides a three-term minimization subspace conjugate gradient optimization method, which specifically includes the following steps:

[0054] Step 1: Given the initial iteration point, the parameters involved in the termination condition, Wolf e Line search parameters, calculate the initial gradient of the function f(x), and determine the initial search direction:

[0055] Step 1.1, set the initial iteration point x0 = (-1.2, 1), set the three parameters of Himmeblau termination condition to ∈1 = ∈2 = 10 -5 ,∈3=10 -6 , set the maximum number of iterations to max_iterations = 15000, and initialize the number of iterations k = 0.

[0056] Step 1.2: Set the two parameters of the Wolfe line search to ρ = 0.01 and σ = 0.72.

[0057] Step 1.3: Calculate the initial gradient The initial direction d0 is set to -g0.

[0058] Step 2: If the function value or gradient satisfies the Himmeblau termination condition, or the number of iterations reaches max_iterations, the iteration is terminated, otherwise go to step 3:

[0059] Step 2.1: Calculation

[0060]

[0061] Step 2.2, if Δ<∈1 or ||g k || ∞ ≤∈3, or the number of iterations reaches max_iterations, the algorithm terminates the iteration and returns the decision variable x k and the number of iterations k, x k is the iteration point of step k, and the current system time is recorded as the end time t1 of the algorithm. The end time is subtracted from the start time t0 to obtain the CPU running time t of the algorithm. CPU =t1-t0. Otherwise, go to step 3.

[0062] Step 3: Take the search step length α k Satisfy the Wolfe line search criterion shown in formula (3):

[0063]

[0064] Step 4: Determine whether to execute the Andrei acceleration strategy and update the decision variables:

[0065] Step 4.1. Calculation where y k =g k+1 -g k ;

[0066] Step 4.2: If Then implement the Andrei acceleration strategy and calculate the acceleration parameters And update the decision variables, that is, let x k+1 =x k +ζ k α k d k ; Otherwise, let x k+1 =x k +α k d k ;

[0067] Step 5: Determine the three-term subspace Ω k+1 The dimension dim k+1 :

[0068] Step 5.1: Calculate s in sequence k =x k+1 -x k ,y k =g k+1 -g k ,y k-1 =g k -g k-1 ;

[0069] Step 5.2: Find the three-term subspace Ω k+1 ={s k ,y k ,y k-1 The dimension dim of k+1 ;

[0070] Step 6: Based on the three-term subspace Ω in step 5 k+1 The dimension dim k+1 Calculate the search direction d for the next iteration k+1 :

[0071] Step 6.1, if dimΩ k+1 =1, then d k+1 =-g k+1 ;

[0072] Step 6.2, if dimΩk+1 =2 and s k With y k Linearly independent, then d k+1 =μ k s k +v k y k , where μ k , v k Determined by formula (4)

[0073]

[0074] Step 6.3, if dimΩ k+1 =2 and s k With y k-1 Linearly independent, then d k+1 =μ k s k +τ k y k-1 , where μ k , τ k Determined by formula (5)

[0075]

[0076] Step 6.4, if dimΩ k+1 =3, then d k+1 =μ k s k +v k y k +τ k y k-1 , where μ k , v k , τ k Determined by formula (6)

[0077]

[0078]

[0079] in, ξ is a constant greater than 1.

[0080] Step 7: Execute the negative gradient restart strategy. Then d k+1 Reset to -g k+1 Then go to step 8, otherwise go directly to step 8.

[0081] Step 8: Accumulate the number of iterations, set k=k+1, and go to step 2.

[0082] In order to verify the effectiveness of the three-term minimization subspace conjugate gradient optimization method (denoted as OURS) proposed in the present invention, the present invention uses Python3.11 and Pytorch2.0.1 (cuda11.8) to test the OURS algorithm on 102 specific optimization problems. Table 1 lists the function names of these optimization problems, where the dimension of each problem is 10000.

[0083] Table 1 Classic unconstrained optimization problems

[0084]

[0085]

[0086] During the test, record the number of iterations N of the OURS algorithm iters , function value calculation times N f And the CPU running time T CPU There are 3 indicators in total, which are used to compare with several classic and effective optimization algorithms such as CG_DESCENT and CGOPT. Some test results are shown in Table 2. It can be seen that in most optimization problems, the OURS algorithm usually has fewer iterations, function value calculations, and CPU running time than the classic CG_DESCENT and CGOPT algorithms. This shows that the OURS algorithm is more efficient, consumes less computing resources, and is more usable.

[0087] Table 2 N of three algorithms in some optimization problems iters 、N f and T CPU contrast

[0088]

[0089]

[0090] Among them, inf means that the maximum number of iterations has been exceeded and there is no convergence.

[0091] At the same time, in order to improve the accuracy of the evaluation, the performance curve proposed by Dolan and Moré is used to compare and evaluate the above three algorithms. CPU For example, suppose To include n p A collection of test questions, is a set of three algorithms, t p,s For the algorithm Solving test problems The CPU running time spent in the process, then the performance ratio of algorithm s to solve the test problem p is defined as

[0092]

[0093] If algorithm s cannot solve the test problem p, then r p,s =+∞, further define the following performance curve

[0094]

[0095] Where size represents the collection Internal satisfaction p,s <The number of test questions of τ, p s (τ) is the cumulative distribution function of the performance ratio of algorithm s. The performance curve of algorithm s can be obtained by sampling τ.

[0096] Sample τ with a step size of 0.1, and get the number of iterations N iters 、N number of function value calculations f And the CPU running time T CPU The performance curves of these three indicators are shown in Figures 2 to 4 Overall, the OURS algorithm has good performance in all three indicators. Specifically, Figure 2 As shown in , OURS solves about 77% of the test problems with the fewest iterations, while CG_DESCENT and CGOPT only solve about 35% of them. Figure 3 As shown in Figure 2, OURS solves about 58% of the test problems with the least number of function value calculations, far ahead of CG_DESCENT's 33% and CGOPT's 38%. Figure 4 As shown in Figure 2, OURS solves about 42% of the test problems in the shortest time, while CG_DESCENT and CGOPT only solve about 30%. Figures 2 to 4 In the figure, the three indicator performance curves of OURS are always at the top, which shows that it is more efficient and stable than the CG_DESCENT and CGOPT algorithms.

[0097] Finally, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A three-term minimization subspace conjugate gradient optimization method, characterized in that: The method comprises the following steps: S1: Take the initial iteration point x0, the initial iteration number is recorded as 0, and calculate the initial gradient of the function f(x) And determine the initial search direction d0 = -g0; S2: If the function value or gradient of the function f(x) satisfies the Himmeblau termination condition, or the number of iterations reaches the set value epoch, the iteration is terminated and the decision variable is returned, otherwise the next step is executed; S3: Determine the search step size based on the Wolfe line search criterion; S4: Determine whether to implement the Andrei acceleration strategy based on the search direction and the search step size in step S3, and update the decision variables; S5: Calculate the dimension of the three-term subspace; S6: Based on the dimensions of the three-term subspace in step S5, calculate the search direction for the next iteration; S7: execute the negative gradient restart strategy; S8: The number of iterations is increased by 1, and the process returns to step S2.

2. A three-term minimization subspace conjugate gradient optimization method according to claim 1, characterized in that: The step S2 comprises the following sub-steps: S21: Order Among them, x k is the iteration point of step k; S22: Record the gradient in the k-th iteration like or || g k || ∞ If ≤∈3 or the number of iterations reaches epoch, the iteration terminates and returns the decision variable, otherwise execute the next step; where ∈1, ∈2, ∈3 are the three parameters of the Himmeblau termination condition.

3. A three-term minimization subspace conjugate gradient optimization method according to claim 1, characterized in that: The step S3 specifically calculates the search step length α based on the gradient and search direction in step S1. k To meet Among them, s k =x k+1 -x k , d k is the search direction of the kth step, ρ and σ are two parameters of the Wolfe line search criterion, and 0<ρ≤σ<1.

4. A three-term minimization subspace conjugate gradient optimization method according to claim 1, characterized in that: The step S4 comprises the following sub-steps: S41: Calculation where yk = gk +1 -gk; S42: If Then implement the Andrei acceleration strategy and calculate the acceleration parameters And update the decision variables, that is, let x k+1 =x k +ζ k α k d k , otherwise x k+1 =x k +α k d k .

5. A three-term minimization subspace conjugate gradient optimization method according to claim 1, characterized in that: The three subspaces in step S5 are composed of s k =x k+1 -x k ,y k =g k+1 -g k and k-1 =g k -g k-1 Spanning the linear subspace Ω k+1 , and the dimension of this subspace is dimΩ k+1 =3, 2 or 1.

6. A three-term minimization subspace conjugate gradient optimization method according to claim 1, characterized in that: The step S6 comprises the following sub-steps: S61: Assumption d k+1 =μ k s k +v k y k +τ k y k-1 , where the coefficient To be determined; S62: D k+1 =μ k s k +v k y k +τ k y k-1 , substitute into the minimization problem , solve for μ k , v k , τ k ; Among them B k+1 is the function f(x) at x k+1 The approximation of the Hessian matrix, assuming that B k+1 is symmetric positive definite and satisfies the quasi-Newton condition, that is, B k+1 s k =y k ; S63: If dimΩ k+1 =1, then d k+1 =-g k+1 ; If dimΩ k+1 =2 and s k With y k Linearly independent, then d k+1 =μ k s k +v k y k ,in If dimΩ k+1 =2 and s k With y k-1 Linearly independent, then d k+1 =μ k s k +τ k y k-1 ,in If dimΩ k+1 =3, then d k+1 =μ k s k +v k y k +τ k y k-1 ,in 7. A three-term minimization subspace conjugate gradient optimization method according to claim 6, characterized in that: In step S63, if dimΩ k+1 =2 and s k With y k Linearly independent, to ensure A k Positive, take Therefore If dimΩ k+1 =2 and s k With y k-1 Linear independence, to ensure Positive, take Therefore If dimΩ k+1 =3, to ensure Positive, take in ξ is a constant greater than 1, so 8. A three-term minimization subspace conjugate gradient optimization method according to claim 1, characterized in that: The negative gradient restart strategy in step S7 is: Then d k+1 Reset to -g k+1 Then execute step S8, otherwise execute step S8 directly.