Self-adaptive multi-order cooperative non-convex optimization control method, device and system

Abstract

The invention provides a self-adaptive multi-order cooperative non-convex optimization control method, device and system. According to the method, rapid convergence is realized by adopting a local optimization strategy based on gradient information, and second-order curvature analysis is automatically introduced after a first-order stable point is reached; and if the second-order stability condition is met, activating high-order information as required, and constructing directional disturbance in the identified escapable subspace to break through stagnation. The above process is iteratively executed until the effective escape direction cannot be identified under the current information. According to the method, on the premise that the calculation overhead of the main optimization stage is not remarkably increased, the capacity of the algorithm for escaping various flat saddle points (including degraded saddle points and weak positive curvature saddle points) and converging to a better local solution is remarkably improved, and the calculation efficiency and the solution quality are effectively balanced.

Description

Technical Field

[0001] This invention relates to the fields of computational mathematics and artificial intelligence, and in particular to an adaptive multi-order cooperative nonconvex optimization control method, device, system, and a method for solving nonconvex optimization problems. Background Technology

[0002] Nonconvex optimization problems are widely found in fields such as artificial intelligence, scientific computing, and engineering decision-making, including deep neural network training, molecular conformation search, and financial portfolio optimization. The objective functions of these problems are usually structurally complex, containing a large number of saddle points, and a significant portion of these saddle points are difficult to escape effectively by low-order optimization methods due to their highly flat local curvature (such as degenerate saddle points or weakly positive curvature saddle points with all positive Hessian but significant third-order effects).

[0003] Specifically, first-order methods (such as gradient descent and Adam) rely solely on gradient information and cannot perceive further optimization directions in flat regions where the gradient is close to zero, easily leading to long-term stagnation. Second-order methods (such as Newton's method and trust regions) can use the Hessian matrix to identify negative curvature directions to escape typical saddle points, but at the aforementioned flat saddle points, the Hessian lacks effective negative eigenvalues, causing its quadratic model to fail to provide a reliable descent path. Introducing higher-order derivative information (such as third-order tensors) can capture the changing trends of function curvature, revealing higher-order escape directions hidden in flat regions, thus overcoming the limitations of lower-order methods and finding higher-quality solutions. However, continuously calculating and storing higher-order tensors throughout the optimization process will result in unbearable computational and memory overhead, limiting its application in practical scenarios.

[0004] In summary, how to effectively identify and escape flat saddle points (including degenerate saddle points and weakly positive curvature saddle points) that are difficult for low-order methods to handle while ensuring computational efficiency is an urgent problem to be solved. Summary of the Invention

[0005] The technical problem this invention aims to solve is that low-order optimization methods in related technologies struggle to effectively escape saddle points with locally flat curvature (such as degenerate saddle points or weakly positive curvature saddle points with all positive Hessian but significant third-order effects), resulting in insufficient convergence accuracy and reliability. Higher-order optimization methods, while possessing higher discriminative power, suffer from high computational complexity and resource consumption, making them difficult to apply efficiently in real-world scenarios. To address these issues, this invention provides an adaptive multi-order cooperative non-convex optimization control method, device, system, and a method for solving non-convex optimization problems.

[0006] According to a first aspect of the present invention, the adaptive multi-order cooperative non-convex optimization control method provided by the present invention includes:

[0007] Step S101: Obtain the objective function and perform an iterative search on the non-convex surface of the objective function until a first-order stable point that satisfies the preset first-order stability condition is obtained;

[0008] Step S102: Based on the first-order stable point, calculate the second-order curvature information corresponding to the first-order stable point, and determine the second-order stability condition to obtain the second-order stable point that satisfies the preset second-order stability condition.

[0009] Step S103: Based on the preset saddle point discrimination model, perform saddle point escapeability discrimination on the second-order stable point, and when the discrimination result is escapeable, determine the corresponding escapeable subspace;

[0010] Step S104: In response to the escapeability determination result of the saddle point being escapeable, based on the escapeable subspace and the third derivative tensor at the second-order stable point, generate and verify a directional perturbation in the escapeable subspace;

[0011] Step S105: If the directional perturbation is verified to be effective, then the new point after the perturbation is used as input, and steps S101 to S104 are repeated until the result of the saddle point escapeability determination is that it is not escapeable.

[0012] Optionally, the step of obtaining the objective function and performing an iterative search on the non-convex surface of the objective function until a first-order stable point satisfying the preset first-order stability condition is obtained includes:

[0013] Based on a preset search model, an iterative search is performed on the non-convex domain of the objective function until the preset first-order stability condition is met, and a first-order stable point is obtained. The search model contains any local optimization algorithm based on gradient information, including but not limited to gradient descent, momentum method, adaptive moment estimation method, and trust region algorithm.

[0014] The first-order stability condition includes: the gradient norm at the current iteration point is less than or equal to a first preset termination threshold; or, when using the trust region algorithm, the estimated decrease of the first-order Taylor model at the current iteration point within the trust region is less than or equal to the product of the first preset termination threshold and the radius of the trust region.

[0015] Optionally, the step of calculating the second-order curvature information corresponding to the first-order stable point and determining the second-order stability condition to obtain a second-order stable point that satisfies the preset second-order stability condition includes:

[0016] Calculate the Hessian matrix at the first-order stable point as second-order curvature information;

[0017] Determine whether the preset second-order stability condition is met;

[0018] If the second-order stability condition is met, the current point is determined as a second-order stable point, and step S103 is executed.

[0019] If any of the second-order stability conditions are not met, then based on the second-order curvature information, it is determined whether to update the current iteration point and / or algorithm parameters, and the current iteration point and corresponding algorithm parameters are fed back to step S101 to continue the iterative process of steps S101 to S102 until a point that satisfies the second-order stability conditions is obtained.

[0020] The second-order stability condition includes: the minimum eigenvalue of the Hessian matrix at the first-order stable point is greater than or equal to a negative second preset termination threshold; or when the trust region algorithm is used, the estimated decrease of the second-order Taylor model at the first-order stable point within the trust region is less than or equal to the product of the second preset termination threshold and the square of the trust region radius.

[0021] Optionally, the step of determining the saddle point escapeability of the second-order stable point based on a preset saddle point discrimination model includes:

[0022] The Hessian matrix at the second-order stable point is decomposed into eigenvalues ​​to obtain multiple unit orthogonal eigenvectors sorted in descending order of eigenvalues.

[0023] The spaces spanned by the i-th to n-th eigenvectors are successively used as candidate subspaces, where i = 1, 2, ..., n, and n is the dimension of the Hessian matrix;

[0024] Calculate the projection norm of the third derivative tensor in the current candidate subspace and determine whether the escape condition is satisfied;

[0025] If the conditions are met, the current point is determined to be an escapeable saddle point, the subspace is recorded as an escapeable subspace, the escapeable subspace and the projection norm of the third derivative tensor in the escapeable subspace are output, and the discrimination is terminated.

[0026] If none of the candidate subspaces are satisfied after traversing all of them, the current point is determined to be an inescapable local optimum, and the optimization process is terminated.

[0027] Optionally, the escape condition includes: the square of the projection norm is greater than or equal to an escape threshold jointly determined by the third-order Lipschitz constant estimate, the eigenvalue of the current candidate subspace initial feature vector, and preset control parameters; the third-order Lipschitz constant estimate has a preset initial value.

[0028] Optionally, the step of generating and verifying the directional perturbation within the escapeable subspace includes:

[0029] Search for a unit vector in the escapeable subspace such that the third action value of the third derivative tensor on the unit vector is greater than or equal to the ratio of the projection norm to the preset control parameter;

[0030] Based on the unit vector, the projection norm, the estimated value of the third-order Lipschitz constant, and the preset control parameters, the directional perturbation step size is calculated and the third-order perturbation step is constructed.

[0031] Based on the third-order perturbation step, calculate the actual function decrease after the perturbation is performed;

[0032] Based on the projection norm, the estimated value of the third-order Lipschitz constant, and the preset control parameters, the theoretical third-order descent reference quantity is calculated.

[0033] Calculate the ratio of the actual function descent to the theoretical third-order descent reference value;

[0034] If the ratio is greater than or equal to the preset acceptance threshold, the new point after the disturbance is accepted, and the new point is returned as input to step S101;

[0035] If the ratio is less than the preset acceptance threshold, the current second-order stable point is kept unchanged, the estimated value of the third-order Lipschitz constant is increased, and the process returns to step S103.

[0036] Optionally, step S101 is implemented using an adaptive trust region framework. This framework dynamically selects either a first-order Taylor model or a second-order Taylor model for constructing the trust region subproblem based on the current local geometric characteristics during the iteration process, and always uses the first-order stability condition as the termination criterion.

[0037] Optionally, the adaptive multi-order cooperative non-convex optimization control method is implemented using an adaptive trust region framework, which switches between a first-order Taylor model and a second-order Taylor model according to the convergence state of the current iteration point.

[0038] According to a second aspect of the present invention, the adaptive multi-order cooperative non-convex optimization control device provided by the present invention includes:

[0039] A first-order solver is used to obtain the objective function and perform iterative search on the non-convex surface of the objective function until a first-order stable point that satisfies the preset first-order stability condition is obtained.

[0040] A second-order solver is used to receive the first-order stable point, calculate the second-order curvature information corresponding to the point, and determine the second-order stability condition to obtain a second-order stable point that satisfies the preset second-order stability condition.

[0041] The saddle point discriminator is used to determine the saddle point escapeability of the second-order stable point according to a preset saddle point discrimination model, and to determine the corresponding escapeable subspace when the discrimination result is escapeable.

[0042] The third-order perturbator includes a perturbation generation unit and a perturbation verification unit. The perturbation generation unit is used to generate a directional perturbation step size based on the third-order derivative tensor at the second-order stable point within the escapeable subspace. The perturbation verification unit is used to verify the effectiveness of the directional perturbation and output the new point after perturbation when the verification is effective.

[0043] The perturbed new point is configured to return to the first-order solver to iteratively execute the aforementioned modules until the result of the saddle point escapeability determination is that it is not escapeable.

[0044] According to a third aspect of the present invention, the present invention provides an adaptive multi-order cooperative non-convex optimization control system main controller, a cross-order information multiplexing bus, and an adaptive multi-order cooperative non-convex optimization control device as described above. The main controller includes a state trigger for controlling the state switching, data flow, and iterative loop between the first-order solver, the second-order solver, the saddle point discriminator, and the higher-order disturbance, forming a closed-loop optimization control logic of low-order optimization → high-order analysis → intelligent escape → feedback loop; the adaptive multi-order cooperative non-convex optimization control device is communicatively connected to the main controller; the cross-order information multiplexing bus is used for data transmission.

[0045] According to a fourth aspect of the present invention, the method for solving nonconvex optimization problems provided by the present invention can employ the adaptive multi-order cooperative nonconvex optimization control method described above to optimize the objective function of the nonconvex optimization problem in order to obtain a solution to the problem.

[0046] In summary, the adaptive multi-order cooperative nonconvex optimization control method, device, system, and a method for solving nonconvex optimization problems provided by this invention have the following advantages compared with related technologies:

[0047] The adaptive multi-order collaborative nonconvex optimization control method described in this invention constructs an adaptive collaborative mechanism of "low-order efficient convergence - high-order accurate escape," which intelligently schedules derivative information of different orders during the nonconvex optimization process: a local optimization strategy dominated by gradient information achieves rapid convergence, and second-order curvature analysis is automatically introduced after reaching a first-order stable point; if the second-order stability condition is met, higher-order information is activated as needed, and directional perturbations are constructed in the identified escapeable subspace to break through stagnation. The above process is executed iteratively until no effective escape direction can be identified under the current information. Its core beneficial effect is that, without significantly increasing the computational overhead of the main optimization stage, it significantly improves the algorithm's ability to escape various flat saddle points (including degenerate saddle points and weakly positive curvature saddle points) and converge to a better local solution, effectively balancing computational efficiency and solution quality.

[0048] It should be noted that the technical solution protected by this invention was completed with funding from the Shanghai Natural Science Foundation (No. 23ZR1445900). Attached Figure Description

[0049] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0050] Figure 1 This is a flowchart illustrating the implementation of an adaptive multi-order cooperative non-convex optimization control method provided by the present invention.

[0051] Figure 2 This is a structural block diagram of an adaptive multi-order cooperative non-convex optimization control device provided in an embodiment of the present invention;

[0052] Figure 3 This is an overall control flowchart of a non-convex optimization control device based on adaptive multi-order cooperative control provided in one embodiment;

[0053] Figure 4 This is a schematic diagram of the structure of an adaptive multi-order cooperative non-convex optimization control system provided in an embodiment of the present invention;

[0054] Figure 5 This is a test experiment result diagram provided in one embodiment. Detailed Implementation

[0055] The core of this invention is to provide an adaptive multi-order cooperative non-convex optimization control method, device, system, and solution method for non-convex optimization problems, which can significantly reduce the computational overhead of all high-order methods while avoiding traditional low-order methods from getting stuck in suboptimal critical points such as saddle points.

[0056] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Please refer to Figure 1 , Figure 1 The flowchart illustrates the implementation of an adaptive multi-order cooperative non-convex optimization control method provided by this invention; the specific operation steps are as follows:

[0058] Step S101: Obtain the objective function and perform an iterative search on the non-convex surface of the objective function until a first-order stable point that satisfies the preset first-order stability condition is obtained;

[0059] Step S102: Based on the first-order stable point, calculate the second-order curvature information corresponding to the first-order stable point, and determine the second-order stability condition to obtain the second-order stable point that satisfies the preset second-order stability condition.

[0060] Step S103: Based on the preset saddle point discrimination model, perform saddle point escapeability discrimination on the second-order stable point, and when the discrimination result is escapeable, determine the corresponding escapeable subspace;

[0061] Step S104: In response to the escapeability determination result of the saddle point being escapeable, based on the escapeable subspace and the third derivative tensor at the second-order stable point, generate and verify a directional perturbation in the escapeable subspace;

[0062] Step S105: If the directional perturbation is verified to be effective, then the new point after the perturbation is used as input, and steps S101 to S104 are repeated until the result of the saddle point escapeability determination is that it is not escapeable.

[0063] For ease of understanding, the following terms are defined in this invention:

[0064] A "first-order stable point" is an iteration point that satisfies at least one of the following first-order stability conditions:

[0065] (a) The gradient norm of the current iteration point is less than or equal to the first preset termination threshold;

[0066] (b) When the trust region algorithm is used, the estimated decrease of the first-order Taylor model at the current iteration point within the trust region is less than or equal to the product of the first preset termination threshold and the radius of the trust region.

[0067] That is, the first-order stability condition includes: the gradient norm of the current iteration point is less than or equal to the first preset termination threshold; or, when the trust region algorithm is used, the estimated decrease of the first-order Taylor model at the current iteration point in the trust region is less than or equal to the product of the first preset termination threshold and the radius of the trust region.

[0068] A "second-order stable point" is an iteration point that satisfies at least one of the following second-order stability conditions:

[0069] (c) The minimum eigenvalue of the Hessian matrix at the first-order stable point is greater than or equal to a negative second preset termination threshold;

[0070] (d) When the trust region algorithm is used, the estimated decrease of the second-order Taylor model constructed at the first-order stable point in the trust region is less than or equal to the product of the second preset termination threshold and the square of the trust region radius.

[0071] In other words, the second-order stability conditions include: the minimum eigenvalue of the Hessian matrix at the first-order stable point is greater than or equal to a negative second preset termination threshold; or when the trust region algorithm is used, the estimated decrease of the second-order Taylor model at the first-order stable point within the trust region is less than or equal to the product of the second preset termination threshold and the square of the trust region radius.

[0072] The above conditions (a) and (b), (c) and (d) are equivalent in their respective applicable scenarios.

[0073] It should be noted that the "second-order stable point" described in this invention is not the second-order stable point referred to in traditional non-convex optimization theory (i.e., a local minimum candidate point that requires zero gradient and a positive semi-definite Hessian matrix), but rather a candidate point that satisfies both first-order and second-order stability conditions, but still requires escape criterion through third-order or higher-order derivative information.

[0074] In some embodiments, the step of obtaining the objective function and performing an iterative search on the non-convex surface of the objective function until a first-order stable point satisfying a preset first-order stability condition is obtained includes:

[0075] Based on a preset search model, an iterative search is performed on the non-convex domain of the objective function until the preset first-order stability condition is met, and a first-order stable point is obtained. The search model contains any local optimization algorithm based on gradient information, including but not limited to gradient descent, momentum method, Adam, and trust region algorithm.

[0076] The first-order stability condition includes: the gradient norm at the current iteration point is less than or equal to a first preset termination threshold; or, when using the trust region algorithm, the estimated decrease of the first-order Taylor model at the current iteration point within the trust region is less than or equal to the product of the first preset termination threshold and the radius of the trust region.

[0077] Specifically, step S101 of the adaptive multi-order cooperative non-convex optimization control method can employ algorithms such as gradient descent (GD), momentum method, adaptive moment estimation (Adam), and trust region algorithm for iterative search. Among these, determining whether the gradient norm of the current iteration point is lower than the first preset termination threshold (the first-order stability condition (a) mentioned above) is one of the more intuitive ways to determine the first-order stable point.

[0078] In other words, when step S101 uses gradient descent, momentum, or adaptive moment estimation for iterative search, the first-order stability condition can be set to the gradient norm at the current iteration point being less than or equal to a first preset termination threshold. When step S101 uses the trust region algorithm for iterative search, the first-order stability condition can be set to the estimated descent of the first-order Taylor model at the current iteration point within the trust region being less than or equal to the product of the first preset termination threshold and the radius of the trust region.

[0079] In another embodiment, the search model employs a trust region algorithm that iterates by solving local Taylor expansion subproblems. It should be noted that in this embodiment, the local model (first-order or second-order) used to generate candidate descent steps is independent of the first-order stability condition used to determine algorithm termination: regardless of the model used for the search, step S101 uses the estimated descent amount of the first-order Taylor model as the termination criterion.

[0080] Example A: When the search model is based on the trust region method of the first-order Taylor model, the specific steps are as follows:

[0081] S1011A: Minimize the first-order Taylor model constructed at the current iteration point within the current trust region to obtain candidate descent steps, and calculate the estimated descent amount of the first-order Taylor model within the trust region;

[0082] S1012A: If the estimated decrease of the first-order Taylor model in the trust region is less than or equal to the product of the first preset termination threshold and the current trust region radius, then the current iteration point is determined as a first-order stable point, and the first-order stable point and its corresponding trust region radius are output to step S102; otherwise, step S1013A is executed.

[0083] S1013A: Calculate the ratio of the actual function descent to the predicted descent corresponding to the first-order Taylor model; if the ratio is greater than or equal to the trust region acceptance threshold, accept the descent step and update the current iteration point, while adaptively adjusting the trust region radius according to the magnitude of the ratio; otherwise, reject the descent step, keep the current iteration point and reduce the trust region radius; return to S1011A to continue the iterative search.

[0084] The trust region acceptance threshold is a parameter used in the trust region algorithm to determine the effectiveness of the descent step.

[0085] In this embodiment, when the search model adopts a first-order Taylor model, the minimization of the model form within the trust region can be expressed by the following formula:

[0086] Formula 1

[0087] In the formula: k is the iteration number index, The current iteration point after the k-th iteration. Let represent the trust region radius at the k-th iteration, and s be the displacement vector (descent step). For the original objective function f at point The value at that location, This indicates that the objective function f is at the point gradient at, Indicates at point The first-order Taylor model is constructed at the location.

[0088] Based on the minimum value of the above model, calculate the estimated decrease in prediction of the first-order Taylor model within the confidence region:

[0089] Formula 2

[0090] The estimated decrease has an analytical form: The computational cost is negligible.

[0091] Further, determine the current iteration point. Whether the first-order stability condition is met. The first-order stability condition can be that the estimated decrease of the first-order Taylor model within the trust region is less than or equal to the product of a first preset termination threshold and the current trust region radius. This first-order stability condition can be expressed by the following formula:

[0092] Formula 3

[0093] In the formula: This is the first preset termination threshold in this example. In other embodiments, the termination threshold may be another positive function of the trust region radius (e.g., ,in ), as long as the function approaches zero with the trust region radius ( The gradient approaches zero to ensure that the algorithm can terminate when the gradient is sufficiently small.

[0094] When the first-order stability condition is satisfied, S101 terminates and outputs the obtained first-order stable point. and its corresponding trust region radius , as the initial input of S102.

[0095] When the first-order stability condition is not met, calculate the ratio of the actual decrease in the function to the decrease predicted by the first-order Taylor model:

[0096] Formula 4

[0097] In the formula: To minimize the first-order descent step obtained from the first-order Taylor model subproblem within the trust region, This represents the actual decrease in the function. The predicted decrease is given by the first-order Taylor model.

[0098] It should be noted that, since the first-order Taylor model satisfies ,and For trust within the domain The smallest descent step, therefore in formula (4) As defined in formula (2) Completely equivalent.

[0099] If the ratio of the actual function decrease to the decrease predicted by the first-order Taylor model is greater than or equal to the trust region acceptance threshold ( ),Right now: If so, accept the descent step and update the iteration point to... Otherwise, reject the descent step and let Furthermore, the trust region radius is reduced to enhance the accuracy of the local model.

[0100] Given the acceptance of the descent step, further based on the ratio With trust region expansion threshold (in The relationship between the two factors is used to adaptively update the trust region radius:

[0101] like Then expand the trust region radius; if If so, maintain or appropriately adjust the trust region radius; if This reduces the trust region radius.

[0102] Then, return to sub-step S1011A and continue the first-order iterative search until the first-order stability condition is met.

[0103] Example B: When the search model uses the trust region method based on the second-order Taylor model to execute step S101, the specific steps include:

[0104] S1011B: Minimize the second-order Taylor model constructed at the current iteration point within the current trust region to obtain candidate descent steps, and calculate the estimated descent amount of the second-order Taylor model within the trust region.

[0105] S1011B': Simultaneously calculates the estimated decrease of the first-order Taylor model within the trust region. This value is equal to the product of the current gradient norm and the radius of the trust region, has an analytical solution, and the computational cost is negligible.

[0106] S1012B: If the estimated decrease of the first-order Taylor model in the trust region is less than or equal to the product of the first preset termination threshold and the current trust region radius, then the current iteration point is determined as a first-order stable point, and the first-order stable point and its corresponding trust region radius are output to step S102; otherwise, step S1013B is executed.

[0107] S1013B: Calculate the ratio of the actual function descent to the predicted descent corresponding to the second-order Taylor model, and based on the relationship between this ratio and the trust region acceptance threshold and expansion threshold, perform adaptive adjustment of the descent step acceptance / rejection and the trust region radius. The specific operation method is consistent with that described in S1013A of Example A; return to S1011B to continue the iterative search.

[0108] It should be noted that this embodiment is highly consistent with Embodiment A in terms of algorithmic flow structure: both generate descent steps by solving the trust region subproblem, make acceptance / rejection decisions based on the trust region ratio, and employ the same radius adaptive strategy. The only difference between the two lies in the local approximation model used: Embodiment A uses a first-order Taylor model, while this embodiment uses a second-order Taylor model. Nevertheless, both termination criteria are unified based on the predicted decrease in the trust region of the first-order Taylor model, to ensure that the first-order stability condition is met before entering S102.

[0109] In a preferred embodiment, step S101 is implemented using an adaptive trust region framework: in each iteration, based on the local geometric characteristics at the current point (e.g., gradient magnitude, Hessian eigenvalue distribution, or curvature estimation), a first-order Taylor model or a second-order Taylor model is dynamically selected to construct the trust region subproblem; regardless of the model selected, the first-order stability condition shown in formula (3) is used as the termination criterion.

[0110] In some embodiments, step S102 is configured to perform a one-time second-order stability analysis and descent step generation operation on the first-order stable point output in step S101, and does not contain an internal iterative loop. Specifically, in response to reaching the first-order stability condition, the Hessian matrix at the current first-order stable point is calculated, and the corresponding second-order stability condition is selected for determination according to the algorithm type used in step S101.

[0111] If step S101 uses algorithms such as gradient descent, momentum method, or Adam that do not output the trust region radius, then the second-order stability condition is: the minimum eigenvalue of the Hessian matrix is ​​greater than or equal to a negative second preset termination threshold.

[0112] If step S101 uses the trust region algorithm and outputs the effective trust region radius, then the second-order stability condition is: the estimated decrease of the second-order Taylor model at this point within the trust region is less than or equal to the product of the second preset termination threshold and the square of the trust region radius.

[0113] If the selected second-order stability condition is met, the current point is determined as a second-order stable point, and the saddle point escapeability judgment in step S103 is triggered; otherwise, a second-order descent step is generated by combining Hessian information (e.g., along the negative curvature direction corresponding to the minimum eigenvalue), and validity verification is performed.

[0114] In the latter case (i.e., step S101 uses the trust region algorithm and outputs the effective trust region radius), the specific process of step S102 includes the following steps:

[0115] S1021: Obtain the first-order stable point output in step S101 and its corresponding trust region radius;

[0116] S1022: Minimize the second-order Taylor model constructed at the first-order stable point within the trust region to obtain candidate second-order descent steps, and calculate the predicted descent amount of the second-order Taylor model within the trust region.

[0117] S1023: If the estimated decrease of the second-order Taylor model in the trust region is less than or equal to the product of the second preset termination threshold and the square of the trust region radius, then the current point is determined as a second-order stable point, and the saddle point escapeability judgment in step S103 is executed; otherwise, step S1024 is executed.

[0118] S1024: Calculate the ratio of the actual function descent to the descent predicted by the second-order Taylor model; if the ratio is greater than or equal to the trust region acceptance threshold, accept the second-order descent step and update the iteration point, while adaptively adjusting the trust region radius according to the magnitude of the ratio; otherwise, reject the second-order descent step, keep the current iteration point unchanged, and reduce the trust region radius; then, take the current iteration point and the updated trust region radius as input and return to step S101.

[0119] Specifically, in obtaining the first-order stable point output in step S101 and its corresponding trust region radius Then, it is used as the initial input in this embodiment, and the second-order initial iteration point is defined as... The initial trust region radius is .

[0120] based on (That is: the first-order stable point generated in step S101) Construct a second-order Taylor model:

[0121] Formula 5

[0122] In the formula: For the original objective function f at point The value at that location, This indicates that the objective function f is at the point gradient at, This indicates that the objective function f is at the point The second-order Hessian matrix at that location, Let the displacement vector represent the descent step. For ease of description, in this invention, the descent step s generated by step S102 is collectively referred to as the second-order descent step.

[0123] Minimize the second-order Taylor model within the trust region:

[0124] Formula 6

[0125] Obtain the minimum value and second-order descent step of the second-order Taylor model within the trust region.

[0126] Based on the minimum value of the second-order Taylor model mentioned above, calculate the predicted decrease of the second-order Taylor model within the confidence region:

[0127] Formula 7

[0128] Further, judgment (That is, whether the first-order stable point generated in step S101) satisfies the second-order stability condition. The second-order stability condition is that the estimated decrease in the second-order Taylor model constructed at the first-order stable point within the trust region is less than or equal to the product of the second preset termination threshold and the square of the trust region radius, i.e.:

[0129] Formula 8

[0130] in The second preset termination threshold is set for this embodiment.

[0131] like Then determine For a second-order stable point, directly output the second-order stable point. And trust region radius Then proceed to step S103.

[0132] like This indicates that although the current point satisfies the first-order stability condition, there is a potential second-order descent direction. Therefore, the previously obtained second-order descent steps... Perform validity verification, i.e., execute step S1024.

[0133] In some embodiments, for the second-order descent step Validation of effectiveness (validation of directed perturbations) includes the following sub-steps:

[0134] First, determine the second-order descent step. Does the second-order descent condition meet? The second-order descent condition is: the ratio of the actual descent to the descent predicted by the second-order Taylor model is greater than or equal to the trust region acceptance threshold, i.e.:

[0135] Formula 9

[0136] In the formula: The second-order descent step obtained for solving the above problem of minimizing the second-order Taylor expansion model. This represents the actual decrease in the function. The predicted descent for the second-order Taylor model is defined in formula (7). Completely equivalent The trust region acceptance threshold is shared with the trust region acceptance threshold used for the first-order descent step judgment in step S101, in order to simplify hyperparameter configuration and maintain algorithm consistency.

[0137] If the above second-order descent condition is met, then the descent step is considered valid, and the update iteration point is: And adaptively adjust the trust region radius according to the magnitude of the ratio; otherwise, reject the descent step and let Furthermore, the trust region radius is reduced to enhance the accuracy of the local model.

[0138] The method for adaptively adjusting the trust region radius based on the ratio is the same as the trust region update strategy used in step S101, specifically: based on the ratio With trust region acceptance threshold and trust region expansion threshold (in The relationship between the two sides is used to adaptively update the trust region radius, and The same trust region expansion threshold is used in step S101. The specific trust region radius update rule is as follows:

[0139] like Then expand the trust region radius; if If so, maintain or appropriately adjust the trust region radius; if This reduces the trust region radius.

[0140] The new trust region radius obtained after the above adjustments is denoted as... and the updated iteration points and trust region radius As the initial input to the first-order solver, jump back to step S101 and continue the iterative descent.

[0141] It should be noted that since the determination of the stable point is closely related to the radius of the trust region, even if the iteration point has not changed, the adjustment of the radius of the trust region may change the effectiveness of the local model. Therefore, it is still necessary to return to S101 to re-determine the convergence.

[0142] In some embodiments, step S103 (saddle point escapeability determination) includes the following sub-steps:

[0143] S1031: Perform eigenvalue decomposition on the Hessian matrix at the second-order stable point to obtain multiple unit orthogonal eigenvectors sorted in descending order of eigenvalues;

[0144] S1032: The spaces spanned by the i-th to n-th eigenvectors are successively used as candidate subspaces, where i=1,2,…,n and n is the dimension of the Hessian matrix;

[0145] S1033: Calculate the projection norm of the third derivative tensor in the current candidate subspace and determine whether the escape condition is satisfied;

[0146] If the conditions are met, the current point is determined to be an escapeable saddle point, the subspace is recorded as an escapeable subspace, the escapeable subspace and the projection norm of the third derivative tensor in the escapeable subspace are output, and the discrimination is terminated.

[0147] If none of the candidate subspaces are satisfied after traversing all of them, the current point is determined to be an inescapable local optimum, and the optimization process is terminated.

[0148] The escape condition includes: the square of the projection norm is greater than or equal to the escape threshold determined by the third-order Lipschitz constant estimate, the eigenvalue of the current candidate subspace initial feature vector, and preset control parameters; the third-order Lipschitz constant estimate has a preset initial value.

[0149] Furthermore, step S103 includes the following steps before step S1031: based on the second-order stable point output in step S102, obtain the Hessian matrix, the third-order derivative tensor, the current estimated value of the third-order Lipschitz constant (with a preset initial value), and the preset control parameters at that point.

[0150] In one specific embodiment, the saddle point escapeability determination process described in step S103 is used to determine the second-order stable point output in step S102. Is it a saddle point from which escape is possible? This process uses preset control parameters. and the current estimate of the third-order Lipschitz constant. (With a preset initial value) as input, and simultaneously, combined with The Hessian matrix and third derivative tensor information at the point are then processed as follows:

[0151] First, calculate the second-order stability point. Corresponding second-order Hessian matrix and the third derivative tensor ;

[0152] Next, eigenvalue decomposition is performed on the Hessian matrix H to obtain... ,in The eigenvalues ​​are arranged in descending order. It is its corresponding unit orthogonal eigenvector;

[0153] Then, according to the index Iterate through each element in the specified order. Perform the following sub-operations:

[0154] • Define candidate subspace }, that is, the subspace spanned by the i-th and all subsequent eigenvectors;

[0155] • Calculate the projection norm of the third derivative tensor in this subspace ;

[0156] • Check if the conditions for escape are met:

[0157] Formula 10

[0158] If, during the traversal, there exists an i such that the above escape condition is met, then the traversal is terminated immediately, and that subspace is recorded. As an escapeable subspace, and preserving the corresponding projection norm. Subsequently, the output can escape the subspace. and its corresponding projection norm This will be passed as input to step S104 for the generation of directional perturbations.

[0159] Otherwise, if the escape condition is not met for all i, then determine... If the point is an inescapable local optimum, then this point is the final solution of the optimization process, and the optimization process terminates.

[0160] It is worth noting that the escapeable subspace determined in this invention differs fundamentally from traditional saddle point escape methods in its construction mechanism. Traditional methods typically rely solely on the non-positive eigenvalues ​​of the Hessian matrix (i.e., The eigenvectors corresponding to the eigenvalues ​​span a "degenerate subspace" and confine the perturbation within this subspace. In contrast, this invention sequentially spans candidate subspaces (i=1,2,…,n) with the i-th eigenvector and its subsequent vectors in descending order of eigenvalues, and calculates the projection norm of the third derivative tensor onto each subspace. Once a subspace is found to satisfy the escape condition (i.e., the projection intensity is sufficiently large relative to the local curvature and the Lipschitz constant), it is immediately identified as an escapeable subspace.

[0161] This traversal strategy enables the present invention to identify a class of critical cases that traditional methods cannot handle: "weakly positive curvature saddle points" where all Hessian eigenvalues ​​are positive (i.e., satisfying the second-order stability condition), but the third-order nonlinear effects are significant. At such points, there may still be effective third-order descent paths along certain positive curvature directions.

[0162] Therefore, the subspace construction mechanism of this invention does not rely on the prior assumption of "the existence of negative curvature", which significantly expands the identification range of escapeable saddle points and improves the robustness and solution capability of the algorithm in complex non-convex problems.

[0163] It should be noted that, in some embodiments, the present invention achieves the identification of generalized saddle points by using a joint criterion of traversing the constructed candidate subspace and the third-order tensor projection. The above-described S103 step constitutes a complete escapeability discrimination mechanism. In some application scenarios with extremely high numerical robustness requirements, a third-order descent potential assessment based on the first-order gradient can be additionally introduced as an auxiliary filtering method, for example: (where g is the gradient norm).

[0164] In some embodiments, the step of generating and verifying the directional perturbation within the escapeable subspace includes:

[0165] Search for a unit vector in the escapeable subspace such that the third action value of the third derivative tensor on the unit vector is greater than or equal to the ratio of the projection norm to the preset control parameter;

[0166] Based on the unit vector, the projection norm, the estimated value of the third-order Lipschitz constant, and the preset control parameters, the directional perturbation step size is calculated and the third-order perturbation step is constructed.

[0167] Based on the third-order perturbation step, calculate the actual function decrease after the perturbation is performed;

[0168] Based on the projection norm, the estimated value of the third-order Lipschitz constant, and the preset control parameters, the theoretical third-order descent reference quantity is calculated.

[0169] Calculate the ratio of the actual function descent to the theoretical third-order descent reference value;

[0170] If the ratio is greater than or equal to the preset acceptance threshold, the new point after the disturbance is accepted, and the new point is returned as input to step S101;

[0171] If the ratio is less than the preset acceptance threshold, the current second-order stable point is kept unchanged, the estimated value of the third-order Lipschitz constant is increased, and the process returns to step S103.

[0172] In this embodiment, step S104 (generating and verifying directional perturbations within the escapeable subspace) includes the following sub-steps:

[0173] S1041: In response to step S103, determine that the current point is an escapeable saddle point, and obtain the corresponding escapeable subspace and the projection norm of the third derivative tensor in the escapeable subspace. Search for a unit vector in the escapeable subspace such that the value of the third derivative tensor at the point along the third direction of the unit vector is greater than or equal to the ratio of the projection norm to the preset control parameter.

[0174] S1042: Based on the unit vector, the projection norm, the current estimate of the third-order Lipschitz constant, and the preset control parameters, calculate the directional perturbation step size and construct the new point after perturbation;

[0175] S1043: Based on the directional perturbation step size, calculate the actual function descent after the perturbation is performed;

[0176] S1044: Calculate the theoretical third-order descent reference quantity based on the projection norm, the estimated value of the third-order Lipschitz constant, and the preset control parameters;

[0177] S1045: Calculate the ratio of the actual function descent to the theoretical third-order descent reference value; if the ratio is greater than or equal to a preset acceptance threshold, accept the new point after the perturbation and return to step S101 as the new point; if the ratio is less than the preset acceptance threshold, keep the current second-order stable point unchanged, increase the estimated value of the third-order Lipschitz constant, and return to step S103.

[0178] Specifically, the information used in step S104 is the same as that in S103, namely: the current point is the second-order stable point output in step S102. Its corresponding escapeable subspace is , for The third derivative tensor in a subspace The projection norm on is denoted as The current estimate of the third-order Lipschitz constant is... Preset control parameters And the preset acceptance threshold for this step to evaluate whether to receive the perturbation step is... Next, based on the above information, the execution process of S104 will be described in detail.

[0179] The estimated value of the third-order Lipschitz constant is maintained as a global iterative state parameter. S104 can be updated according to the perturbation effect and fed back to the subsequent S103 and S104 steps.

[0180] First, generate perturbation directions with descent potential in the escapeable subspace: in the escapeable subspace Repeatedly sampled standard Gaussian vectors and normalized them to unit vectors This continues until a vector is found that satisfies the following condition:

[0181] Formula 11

[0182] In the formula The third derivative tensor represents the direction of the tensor. The value of the third-order directional derivative on.

[0183] Then, the directional perturbation step size is calculated and a third-order perturbation step is constructed:

[0184] Formula 12

[0185] The new point after the disturbance is .

[0186] Next, the effectiveness of the third-order perturbation step is evaluated. The actual function decrease after the perturbation is calculated. and theoretical third-order descent reference quantity And define the third-order descent evaluation index. The ratio of the two is:

[0187] Formula 13

[0188] if If the third-order perturbation step is valid, the new point after the perturbation is accepted. and will As the initial input for step S101, a new round of optimization iterations for the first-order solver part is initiated; (in the embodiment employing an adaptive trust region framework, the trust region radius output in step S102 is also used). (denoted as the new initial trust region radius) This serves as the initial input for S101.

[0189] if This indicates that the current third-order descent step has failed to bring about a sufficient decrease in the function value, possibly due to an underestimated third-order Lipschitz constant. At this point, we maintain... Keeping the third-order Lipschitz constant unchanged, update the estimated value: (in (This is the preset magnification factor). The updated version will then be... and origin Returning to step S103, the saddle point discrimination and perturbation generation process is executed again. It is worth noting that since the Hessian matrix has not changed, the eigenvalue decomposition result of step S1032 can be reused, and the threshold comparison only needs to be re-executed based on the updated third-order Lipschitz constant estimate.

[0190] In some embodiments, step S105 (optimization termination determination) is not involved in the perturbation effectiveness decision, but is specifically used to capture the global convergence signal. Specifically, it includes the following sub-steps:

[0191] S1051: In response to step S103, if the current second-order stable point is determined to be unavoidable, the entire optimization process is terminated, and the current point is output as the final optimization result.

[0192] It is worth noting that the estimated value of the third-order Lipschitz constant... Maintained as a global iterative state parameter, it is used in conjunction with the saddle point discrimination in step S103 and the perturbation verification in step S104, and is dynamically updated. For saddle points determined to be escapeable, this embodiment constructs a directional perturbation with descent potential within the escapeable subspace identified by the saddle point discriminator to effectively break through the flat region around degenerate saddle points (which are difficult to escape). The perturbation step size is adaptively controlled by the estimated value of the third-order Lipschitz constant: initially using a preset value, and then adjusted according to the ratio of the actual function descent to the theoretical descent reference value during the iteration process. This mechanism does not require precise pre-estimation of the true third-order Lipschitz constant, which reduces computational complexity and achieves a good balance between escape capability and numerical stability. The new point generated by the perturbation will be used as the initial input of the first-order solver to continue executing subsequent iterations, thereby realizing multi-order collaborative non-convex optimization control.

[0193] This embodiment aims to determine the escapeability of the second-order stable points to which the algorithm converges. The core idea is to analyze the projection of the third-order derivative tensor onto the eigenvector subspaces of the Hessian matrix, paying particular attention to directions with low curvature or significant high-order nonlinearity. If the norm of this projection is non-zero, it indicates the possible existence of a valid third-order descent direction. Although the current point satisfies the first and second-order stability conditions, it is still an escapeable saddle point; in this case, a third-order perturbation mechanism will be triggered. Conversely, if the projection norm is zero, it indicates the absence of high-order descent information in the subspace, and it can be determined that a local optimum has been reached, causing the algorithm to terminate iteration. This discrimination strategy activates third-order analysis only when a second-order stable point is detected, avoiding blindly performing high-order calculations on all iteration points, thereby significantly reducing the overall computational cost.

[0194] In summary, the adaptive multi-order collaborative nonconvex optimization control method described in this invention, by constructing an adaptive collaborative mechanism of "low-order efficient convergence – high-order accurate escape," can intelligently schedule derivative information of different orders during the nonconvex optimization process: a local optimization strategy dominated by gradient information achieves rapid convergence, and after reaching a first-order stable point, second-order curvature analysis is automatically introduced; if the second-order stability condition is met, higher-order information is activated as needed, and directional perturbations are constructed in the identified escapeable subspace to break through stagnation. The above process is iteratively executed until no effective escape direction can be identified under the current information. Its core beneficial effect is that, without significantly increasing the computational overhead of the main optimization stage, it significantly improves the algorithm's ability to escape various flat saddle points (including degenerate saddle points and weakly positive curvature saddle points) and converge to a better local solution, effectively balancing computational efficiency and solution quality.

[0195] Please refer to Figure 2 , Figure 2 A structural block diagram of an adaptive multi-order cooperative non-convex optimization control device provided in an embodiment of the present invention; the specific device may include:

[0196] A first-order solver 100 is used to obtain the objective function and perform iterative search on the non-convex surface of the objective function until a first-order stable point that satisfies the preset first-order stability condition is obtained.

[0197] In specific embodiments, gradient descent, momentum method, Adam, trust region algorithm, etc., can be implemented. It has a first-order stable point trigger, which is used to determine whether the triggering condition is met based on the gradient norm or the estimated descent amount from the first-order Taylor model.

[0198] The second-order solver 200 is communicatively connected to the first-order solver 100. It is used to calculate the second-order curvature information corresponding to the first-order stable point based on the first-order stable point, and to determine the second-order stability condition to obtain a second-order stable point that satisfies the preset second-order stability condition.

[0199] In a specific embodiment, after the first-order stability condition is satisfied, the current first-order stable point is received, and its Hessian matrix is ​​calculated as second-order curvature information to determine the second-order stability condition. If the second-order stability condition is satisfied, the second-order stable point is output; if not, the second-order descent direction is calculated based on the Hessian matrix, candidate descent steps are generated, and validity verification is performed; if the verification passes, the current point is updated; otherwise, it remains unchanged; finally, the current point and algorithm parameters are fed back to the first-order solver 100.

[0200] The saddle point discriminator 300 is communicatively connected to the second-order solver 200 and is used to determine the saddle point escapeability of the second-order stable point according to a preset saddle point discrimination model.

[0201] In a specific embodiment, after obtaining the second-order stable point, the saddle point escapeability determination is triggered. Specifically, the Hessian matrix is ​​decomposed into eigenvalues, and candidate subspaces spanned by the i-th and subsequent eigenvectors are constructed in descending order of eigenvalues. The projection of the third derivative tensor onto each candidate subspace is calculated. If a candidate subspace satisfies the preset escapeability condition, it is determined as an escapeable subspace, and the saddle point is determined as an escapeable saddle point based on the relationship between the projection norm and the eigenvalues.

[0202] A third-order perturbator 400, communicatively connected to the saddle point discriminator 300, is used to receive the escapeable subspace determined by the saddle point discriminator 300 in response to the saddle point escapeability determination result being escapeable. Based on the third-order derivative tensor at the second-order stable point, it generates and performs a directional perturbation within the escapeable subspace to obtain a new perturbation point. The new perturbation point is then returned as input to the first-order solver 100 for iterative execution of each module until the saddle point escapeability determination result is non-escapeable.

[0203] Furthermore, the third-order perturbator 400 includes a perturbation generation unit and a perturbation verification unit. The perturbation generation unit is used to generate a directional perturbation step size based on the third-order derivative tensor at the second-order stable point within the escapeable subspace. The perturbation verification unit is used to verify the effectiveness of the directional perturbation and output the new point after perturbation when the verification is effective.

[0204] In a specific embodiment, when the escapeability determination result of the saddle point is escapeable, based on the third derivative information of the second-order stable point, a directional perturbation vector is generated in the escapeable subspace through random sampling and conditional filtering, and an adaptive step size is calculated to perform the perturbation, thereby obtaining a new point after perturbation; it is also used to verify the effectiveness of the perturbation, and if the verification fails, the estimated value of the third-order Lipschitz constant is adaptively adjusted.

[0205] The adaptive multi-order cooperative non-convex optimization control device of this embodiment is used to implement the aforementioned adaptive multi-order cooperative non-convex optimization control method. Therefore, the specific implementation method and corresponding technical effects of the adaptive multi-order cooperative non-convex optimization control device can be found in the embodiment section of the adaptive multi-order cooperative non-convex optimization control method above. For example, the first-order solver 100, the second-order solver 200, the saddle point discriminator 300, and the third-order disturbance 400 are used to implement steps S101-S105 in the above-mentioned adaptive multi-order cooperative non-convex optimization control method. Therefore, its specific implementation method can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0206] In some embodiments, the adaptive multi-order cooperative non-convex optimization control method can be implemented by an adaptive trust region framework, which can automatically switch between a first-order Taylor model and a second-order Taylor model according to the convergence state of the current iteration point.

[0207] The design of this invention has good modularity: in the first-order solver, any local optimization algorithm that can output a first-order stable point (such as gradient descent, Adam, trust region algorithm, etc.) can be used, as long as it satisfies the first-order stability condition as the termination criterion; in the second-order solver, Hessian-based analysis methods (such as trust region subproblem solving, negative curvature direction search, etc.) can be used, as long as it performs a single second-order decision and supports feedback to the first-order solver.

[0208] All adjustments to this implementation method are within the protection scope of this invention and will not disrupt the core collaborative logic of "first-order → second-order → saddle point discrimination → disturbance → feedback".

[0209] like Figure 3 , Figure 3This is an overall control flowchart of an adaptive multi-order cooperative non-convex optimization control device provided in one embodiment. The overall process follows a hierarchical cooperative optimization architecture of "first-order dominant → second-order fine-tuning → third-order escape (on demand)," balancing efficiency and high-order escape capability. It is worth noting that this framework has good flexibility: users can choose different main optimization strategies within the first-order dominant overall framework, depending on the problem characteristics or computational resources—such as using a pure first-order gradient-based algorithm, or using an adaptive trust region method in the S101 instance.

[0210] In some embodiments, the system improves the efficiency of the main optimization by introducing second-order information in S101, and after obtaining the first-order stable point, the second-order solver performs stability determination. If necessary, it combines third-order perturbation to achieve saddle point escape. The final result is generated by this cooperative mechanism.

[0211] Please refer to Figure 4 , Figure 4 This is a schematic diagram of an adaptive multi-order cooperative non-convex optimization control system provided in an embodiment of the present invention; the specific system may include:

[0212] The main controller includes state triggers, which are used to control the state switching, data flow and iterative loop between the first-order solver, second-order solver, saddle point discriminator and third-order perturbator, forming a closed-loop optimization control logic of "low-order optimization → high-order analysis → intelligent escape → feedback loop".

[0213] For example, the new point after the perturbation is used as input and returned to the first-order solver to iteratively execute the steps in the first-order solver, second-order solver, saddle point discriminator and third-order perturbator until the saddle point escapeability judgment result is that it is not escapeable.

[0214] The aforementioned adaptive multi-order cooperative non-convex optimization control device is communicatively connected to the main controller;

[0215] A cross-order information multiplexing bus is used for data transmission.

[0216] This invention also provides a method for solving nonconvex optimization problems, including:

[0217] The objective function of the nonconvex optimization problem is optimized using the adaptive multi-order cooperative nonconvex optimization control method described above, so as to obtain a solution to the problem.

[0218] This embodiment provides a method for solving non-convex optimization problems, including the following steps:

[0219] Problem definition steps: Receive the optimization problem to be solved, which is in the form of... The objective function To define in space (usually) A continuous non-convex function on (or a subset thereof), which may contain multiple local optima, saddle points, and especially degenerate saddle points. The problem originates from any field, including machine learning, operations research, financial engineering, and scientific computing.

[0220] Method application steps: Using the aforementioned adaptive multi-order cooperative nonconvex optimization control method, the objective function of the problem to be optimized is... As the objective function in the method, and given an initial point. .

[0221] Execution and solution steps: Run the adaptive multi-order collaborative non-convex optimization control method. This method will automatically execute its process of "low-order optimization convergence → saddle point discrimination triggering → high-order escape decision and execution → closed-loop feedback", intelligently switching between information of different orders to effectively escape flat regions and degenerate saddle points.

[0222] Output steps: When the method meets its termination condition (such as the saddle point being identified as an inescapable local optimum, or the maximum number of iterations being reached), output the last obtained iteration point as a solution to the non-convex optimization problem, and output the corresponding objective function value.

[0223] Based on the above embodiments, to verify the effectiveness of the present invention in practical nonconvex optimization problems, experiments were conducted on a class of nonconvex logistic regression models based on the sigmoid function. This problem can be applied to binary classification tasks in machine learning, and its objective function is as follows:

[0224] Formula 14

[0225] in For the sample set, labels The regularization parameter is set to The objective function is a combination of squared loss and sigmoid activation function. The objective function is non-convex and may have a large number of degenerate saddle points, which places high demands on the escape capability of the optimization algorithm.

[0226] The main optimization method used in this invention is called the Progressive-Order Nonconvex Optimization Algorithm (PONOA).

[0227] To evaluate the performance advantages of this invention, two methods were selected as comparative baselines:

[0228] (1) The classic second-order trust region algorithm (TR) – without a higher-order escape mechanism;

[0229] (2) A general high-order strategy (denoted as CONOA) that combines third-order information for detection in each iteration step - this method can be regarded as a dense call variant of the present invention, which performs complete third-order escapeability discrimination and perturbation generation at each iteration point, while PONOA only activates the third-order escape module as needed when a second-order stable point is detected.

[0230] In this embodiment, the S101 stage of PONOA adopts the same adaptive trust region framework as TR (mainly based on the second-order Taylor model) to fairly compare the gains brought by higher-order mechanisms.

[0231] All algorithm parameters are set within their theoretically permissible range and in accordance with conventional optimization practices to ensure the stability and convergence of the algorithms. All three methods use the same initial parameter configuration to further ensure the fairness of the experimental comparison.

[0232] The experiment was conducted using two datasets from LIBSVM. For each dataset, 10 initial points following a normal distribution were randomly generated. The specific results are as follows: Figure 5 As shown.

[0233] Experimental results show that when solving non-convex logistic regression problems, this invention activates the third-order escape mechanism only when key geometric regions such as second-order stable points are identified, effectively avoiding the redundant overhead caused by CONOA-type methods that calculate the third-order derivative tensor information in each iteration. Compared with such strategies that intensively call high-order information, this invention significantly reduces computational overhead while maintaining the same solution quality; at the same time, compared with traditional second-order methods (TR), it can more reliably escape degenerate saddle points and converge to a better solution. This fully verifies that this invention achieves a better synergistic balance among computational efficiency, escape capability, and solution quality.

[0234] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An adaptive multi-order cooperative non-convex optimization control method, characterized in that, include: Step S101: Obtain the objective function and perform an iterative search on the non-convex surface of the objective function until a first-order stable point that satisfies the preset first-order stability condition is obtained; Step S102: Based on the first-order stable point, calculate the second-order curvature information corresponding to the first-order stable point, perform second-order stability condition determination, and obtain a second-order stable point that satisfies the preset second-order stability condition. Step S103: Based on the preset saddle point discrimination model, perform saddle point escapeability discrimination on the second-order stable point. When the discrimination result is escapeable, determine the corresponding escapeable subspace. Step S104: In response to the escapeability determination result of the saddle point being escapeable, based on the escapeable subspace and the third derivative tensor at the second-order stable point, generate and verify a directional perturbation in the escapeable subspace; Step S105: If the directional perturbation is verified to be effective, then the new point after the perturbation is used as input, and steps S101 to S104 are repeated until the result of the saddle point escapeability determination is that it is not escapeable.

2. The adaptive multi-order cooperative non-convex optimization control method according to claim 1, characterized in that, The step of obtaining the objective function and performing an iterative search on the non-convex surface of the objective function until a first-order stable point satisfying the preset first-order stability condition is obtained includes: Based on a preset search model, an iterative search is performed on the non-convex domain of the objective function until the preset first-order stability condition is met, and a first-order stable point is obtained. The search model contains any local optimization algorithm based on gradient information, including but not limited to gradient descent, momentum method, adaptive moment estimation method, and trust region algorithm. The first-order stability condition includes: the gradient norm at the current iteration point is less than or equal to a first preset termination threshold; or, when using the trust region algorithm, the estimated decrease of the first-order Taylor model at the current iteration point within the trust region is less than or equal to the product of the first preset termination threshold and the radius of the trust region.

3. The adaptive multi-order cooperative non-convex optimization control method according to claim 1 or 2, characterized in that, The step of calculating the second-order curvature information corresponding to the first-order stable point based on the first-order stable point, and performing a second-order stability condition determination to obtain a second-order stable point that satisfies the preset second-order stability condition includes: Calculate the Hessian matrix at the first-order stable point as second-order curvature information; Determine whether the preset second-order stability condition is met; If the second-order stability condition is met, the current point is determined as a second-order stable point, and step S103 is executed. If any of the second-order stability conditions are not met, then based on the second-order curvature information, it is determined whether to update the current iteration point and / or algorithm parameters, and the current iteration point and corresponding algorithm parameters are fed back to step S101 to continue the iterative process of steps S101 to S102 until a point that satisfies the second-order stability conditions is obtained. The second-order stability condition includes: the minimum eigenvalue of the Hessian matrix at the first-order stable point is greater than or equal to a negative second preset termination threshold; or when the trust region algorithm is used, the estimated decrease of the second-order Taylor model at the first-order stable point within the trust region is less than or equal to the product of the second preset termination threshold and the square of the trust region radius.

4. The adaptive multi-order cooperative non-convex optimization control method according to claim 1, characterized in that, The step of determining the saddle point escapeability of the second-order stable point based on a preset saddle point discrimination model includes: The Hessian matrix at the second-order stable point is decomposed into eigenvalues ​​to obtain multiple unit orthogonal eigenvectors sorted in descending order of eigenvalues. The spaces spanned by the i-th to n-th eigenvectors are successively used as candidate subspaces, where i = 1, 2, ..., n, and n is the dimension of the Hessian matrix; Calculate the projection norm of the third derivative tensor in the current candidate subspace and determine whether the escape condition is satisfied; If the conditions are met, the current point is determined to be an escapeable saddle point, the subspace is recorded as an escapeable subspace, the escapeable subspace and the projection norm of the third derivative tensor in the escapeable subspace are output, and the discrimination is terminated. If none of the candidate subspaces are satisfied after traversing all of them, the current point is determined to be an inescapable local optimum, and the optimization process is terminated.

5. The adaptive multi-order cooperative non-convex optimization control method according to claim 4, characterized in that, The escape condition includes: the square of the projection norm is greater than or equal to the escape threshold determined by the third-order Lipschitz constant estimate, the eigenvalue of the current candidate subspace initial feature vector, and preset control parameters; the third-order Lipschitz constant estimate has a preset initial value.

6. The adaptive multi-order cooperative non-convex optimization control method according to claim 4, characterized in that, The step of generating and verifying the directional perturbation within the escapeable subspace includes: Search for a unit vector in the escapeable subspace such that the third action of the third derivative tensor on the unit vector is greater than or equal to the ratio of the projection norm to the preset control parameter. Based on the unit vector, the projection norm, the estimated value of the third-order Lipschitz constant, and the preset control parameters, the directional perturbation step size is calculated and the third-order perturbation step is constructed. Based on the third-order perturbation step, calculate the actual function decrease after the perturbation is performed; Based on the projection norm, the estimated value of the third-order Lipschitz constant, and the preset control parameters, the theoretical third-order descent reference quantity is calculated. Calculate the ratio of the actual function descent to the theoretical third-order descent reference value; If the ratio is greater than or equal to the preset acceptance threshold, the new point after the disturbance is accepted, and the new point is returned to step S101 as input; If the ratio is less than the preset acceptance threshold, the current second-order stable point is kept unchanged, the estimated value of the third-order Lipschitz constant is increased, and the process returns to step S103.

7. The adaptive multi-order cooperative non-convex optimization control method according to claim 6, characterized in that, Step S101 is implemented using an adaptive trust region framework. During the iteration process, this framework dynamically selects either a first-order Taylor model or a second-order Taylor model for constructing the trust region subproblem based on the current local geometric characteristics, and always uses the first-order stability condition as the termination criterion.

8. An adaptive multi-order cooperative non-convex optimization control device, characterized in that, include: A first-order solver is used to obtain the objective function and perform iterative search on the non-convex surface of the objective function until a first-order stable point that satisfies the preset first-order stability condition is obtained. The second-order solver is used to receive the first-order stable point, calculate the second-order curvature information corresponding to the point, and determine the second-order stability condition to obtain the second-order stable point that satisfies the preset second-order stability condition. The saddle point discriminator is used to determine the saddle point escapeability of the second-order stable point according to a preset saddle point discrimination model, and to determine the corresponding escapeable subspace when the discrimination result is escapeable. The third-order perturbator includes a perturbation generation unit and a perturbation verification unit. The perturbation generation unit is used to generate a directional perturbation step size based on the third-order derivative tensor at the second-order stable point within the escapeable subspace. The perturbation verification unit is used to verify the effectiveness of the directional perturbation and output the new point after perturbation when the verification is effective. The perturbed new point is configured to return to the first-order solver to iteratively execute the aforementioned modules until the result of the saddle point escapeability determination is that it is not escapeable.

9. An adaptive multi-order cooperative non-convex optimization control system, characterized in that, The system includes a main controller, a cross-order information multiplexing bus, and an adaptive multi-order collaborative non-convex optimization control device as described in claim 8. The main controller includes state triggers for controlling the state switching, data flow, and iterative loops between the first-order solver, second-order solver, saddle point discriminator, and third-order perturbator, forming a closed-loop optimization control logic of low-order optimization → high-order analysis → intelligent escape → feedback loop. The adaptive multi-order collaborative non-convex optimization control device is communicatively connected to the main controller. The cross-order information multiplexing bus is used for data transmission.

10. A method for solving nonconvex optimization problems, characterized in that, include: The objective function of the nonconvex optimization problem is optimized using the adaptive multi-order cooperative nonconvex optimization control method as described in any one of claims 1 to 7, so as to obtain a solution to the problem.

Images