Adaptive optimization rapid convergence method based on symbol gradient

Abstract

The invention relates to the technical field of machine learning and unconstrained optimization, and discloses a symbol gradient-based adaptive optimization rapid convergence method. According to the method, for a deterministic optimization scene, an objective function is optimized according to an adaptive symbol gradient descent algorithm; and aiming at the randomness optimization scene, optimizing the target function according to an adaptive symbol stochastic gradient descent algorithm. Wherein the adaptive symbol gradient descent algorithm and the adaptive symbol stochastic gradient descent algorithm both adopt the distance between the current objective function value and the initial objective function value and a preset key hyper-parameter, and the step length is updated in each iteration. The updating of the adaptive step length of the algorithm does not depend on related information of a gradient and a norm thereof, so that the communication overhead between nodes is effectively reduced. And the adaptive symbol stochastic gradient descent algorithm allows random estimation of the gradient to be biased estimation of the real gradient, so that good convergence performance is ensured.

Description

Technical Field

[0001] The present application relates to the technical field of machine learning and unconstrained optimization, and in particular to a fast convergence method for adaptive optimization based on symbolic gradient. Background Art

[0002] With the continuous development of machine learning technology, the optimization problem at its core has become crucial. Many model training tasks in machine learning can be regarded as unconstrained optimization problems. For example, the parameter optimization of models such as linear regression, logistic regression, and neural networks can be achieved by minimizing the loss function.

[0003] Existing classical optimization algorithms, such as gradient descent and stochastic gradient descent, have been proven to be effective in optimizing various objective functions. In order to save communication costs in distributed optimization scenarios and to avoid the gradient explosion and gradient vanishing problems that may exist in classical algorithms, a signed gradient-based optimization algorithm was introduced to compress the communication cost from 32 bits to 1 bit in each dimension. However, in the general case where the objective function is inseparable, the signed gradient-based optimization algorithm often fails to achieve precise convergence, but oscillates repeatedly near the optimal solution, and the adjustment of the oscillation neighborhood lacks flexibility. Therefore, many scholars have conducted further research and improvements on this basis.

[0004] For example, some scholars introduced the step size of the binary form into the symbolic gradient descent algorithm and proposed the DICHO algorithm. Through a large number of experiments, it was shown that the algorithm solves the oscillation problem. However, the algorithm cannot converge accurately in the general case where the objective function is inseparable, and the size of the neighborhood cannot be adjusted. Some scholars have also proposed the Scaled signGD, ScaledsignSGD and EF-signSGD algorithms, which can all show better performance than the traditional symbolic gradient descent method from theoretical proofs and experiments, and can achieve accurate linear convergence even when the objective function is inseparable. However, the use of gradient norms in these algorithms increases the communication burden. In addition, in the random optimization scenario, various gradient-based optimization algorithms and the above-mentioned symbolic gradient-based optimization algorithms all require that the stochastic gradient is an unbiased estimate of the true gradient and the variance is bounded, which is not suitable for many practical application scenarios. Summary of the invention

[0005] The present application provides a fast convergence method for adaptive optimization based on symbolic gradient to solve the problems in the prior art that the convergence effect of the optimization algorithm based on symbolic gradient depends on the separability of the objective function, has high requirements on the unbiasedness of the random estimation of the gradient, and uses the gradient norm in the algorithm iteration, which leads to increased communication costs.

[0006] In order to solve the above technical problems, the present application discloses a fast convergence method for adaptive optimization based on symbolic gradient, the method comprising:

[0007] For deterministic optimization scenarios, the objective function is optimized according to the adaptive symbolic gradient descent algorithm;

[0008] For random optimization scenarios, the objective function is optimized according to the adaptive symbolic stochastic gradient descent algorithm;

[0009] Among them, both the adaptive symbolic gradient descent algorithm and the adaptive symbolic stochastic gradient descent algorithm use the distance between the current objective function value and the initial objective function value and the preset key hyperparameters to update the step size in each iteration;

[0010] The adaptive signed stochastic gradient descent algorithm allows the random estimate of the gradient to become a biased estimate of the true gradient.

[0011] Preferably, the adaptive signed gradient descent algorithm is:

[0012] Determine initial parameters Initial step size α0∈(0,+∞) and key hyperparameters β,γ∈(0,+∞);

[0013] Initialize iteration index k=0;

[0014] Step 1.1:

[0015] Step 1.2:

[0016] Step 1.3: Let k = k + 1, and return to step 1.1;

[0017] in, It means that the objective function f is The gradient at ; sign represents the sign function.

[0018] Preferably, according to the fast convergence method of adaptive optimization based on signed gradient of claim 2, it is characterized in that the basic assumptions of the objective function are: strongly convex and smooth, or non-convex and smooth, but satisfying Polyak- Assumption.

[0019] Preferably, according to the fast convergence method of adaptive optimization based on signed gradient according to claim 3, it is characterized in that, for the deterministic optimization scenario, after optimizing the objective function according to the adaptive signed gradient descent algorithm, the method further comprises:

[0020] Determine smoothness parameters and strong convexity parameters based on the basic assumptions of the objective function;

[0021] Determine the first convergence condition based on key hyperparameters, smoothness parameters, and strong convexity parameters;

[0022] The convergence of the adaptive symbolic gradient descent algorithm is demonstrated based on the first convergence condition.

[0023] Preferably, the method for fast convergence of adaptive optimization based on signed gradient according to claim 1 is characterized in that the adaptive signed stochastic gradient descent algorithm is:

[0024] Determine initial parameters Initial step size α0∈(0,+∞), key hyperparameters β,γ∈(0,+∞);

[0025] Initialize iteration index k=0;

[0026] Step 2.1: Get a random estimate of the gradient

[0027] Step 2.2:

[0028] Step 2.3:

[0029] Step 2.4: Let k = k + 1, and return to step 2.1;

[0030] in, It means that the objective function f is The gradient at ; sign represents the sign function.

[0031] Preferably, according to the fast convergence method of adaptive optimization based on symbolic gradient of claim 5, it is characterized in that, in the adaptive symbolic stochastic gradient descent algorithm, the SPB hypothesis that allows the random estimate of the gradient to be a biased estimate of the true gradient is set:

[0032] For any k ≥ 0 and i ∈ [d], the stochastic estimate of the gradient is With the true gradient Satisfy between:

[0033]

[0034] in, and Respectively and The value of the i-th element of .

[0035] Preferably, according to the fast convergence method of adaptive optimization based on signed gradient according to claim 6, it is characterized in that, for the stochastic optimization scenario, after optimizing the objective function according to the adaptive signed stochastic gradient descent algorithm, the method further comprises:

[0036] Under the premise of satisfying the SPB assumption, define the minimum success probability p min for:

[0037]

[0038] Preferably, according to the fast convergence method of adaptive optimization based on signed gradient according to claim 7, it is characterized in that, under the premise of satisfying the SPB assumption, after defining the minimum success probability, the method further comprises:

[0039] Determine smoothness parameters and strong convexity parameters based on the basic assumptions of the objective function;

[0040] The second convergence condition is determined based on key hyperparameters and smoothness parameters, strong convex parameters, and minimum success probability;

[0041] The convergence of the adaptive symbolic stochastic gradient descent algorithm is demonstrated based on the second convergence condition;

[0042] The basic assumptions of the objective function are: strongly convex and smooth, or non-convex and smooth, but satisfying Polyak- Assumption.

[0043] In this application, for deterministic optimization scenarios, the objective function is optimized according to the adaptive symbolic gradient descent algorithm; for stochastic optimization scenarios, the objective function is optimized according to the adaptive symbolic stochastic gradient descent algorithm. Among them, both the adaptive symbolic gradient descent algorithm and the adaptive symbolic stochastic gradient descent algorithm use the distance between the current objective function value and the initial objective function value and the preset key hyperparameters to update the step size in each iteration. The update of the adaptive step size of the algorithm does not depend on the relevant information of the gradient and its norm, which meets the needs of distributed application scenarios to save communication costs, avoids transmitting complete gradient information, and instead transmits compressed gradient information (sign function of gradient), effectively reducing the communication overhead between nodes. And the adaptive symbolic stochastic gradient descent algorithm allows the random estimate of the gradient to be a biased estimate of the true gradient, ensuring good convergence performance.

[0044] In addition, from a theoretical perspective, this application rigorously proves the linear convergence rate of the algorithm under reasonable assumptions, studies the impact of hyperparameter adjustment on the size of the convergence neighborhood, determines the conditions for precise convergence, and meets the needs of performance analysis of symbolic gradient-based optimization algorithms, that is, ensuring the speed and accuracy of convergence performance during algorithm execution.

[0045] In addition, this application studies the impact of the SPB assumption and the minimum success probability on the AS-SGD algorithm, so that the algorithm can still converge effectively when it does not rely on the strict unbiased estimation assumption, meeting the practical design requirements of stochastic optimization algorithms, that is, the algorithm can still maintain good performance when encountering the common situation in reality where there is a large estimation error in the random estimation of the gradient.

[0046] Additional aspects and advantages of the present application will be given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0048] Figure 1 A flowchart of a fast convergence method for adaptive optimization based on signed gradient provided in an embodiment of the present application;

[0049] Figure 2 The overall technical roadmap of the adaptive optimization algorithm of the signed gradient provided in the embodiments of the present application;

[0050] Figure 3 A simulation result diagram of a deterministic optimization algorithm for the MOP2 function provided in an embodiment of the present application;

[0051] Figure 4 A simulation result diagram (logarithmic form) of a deterministic optimization algorithm for the MOP2 function provided in an embodiment of the present application;

[0052] Figure 5 A simulation result diagram of the random optimization algorithm for the MOP2 function provided in an embodiment of the present application;

[0053] Figure 6 A simulation result diagram (logarithmic form) of the random optimization algorithm for the MOP2 function provided in an embodiment of the present application;

[0054] Figure 7 A simulation result diagram of a deterministic optimization algorithm for a logistic regression function provided in an embodiment of the present application;

[0055] Figure 8 A simulation result diagram (in logarithmic form) of a deterministic optimization algorithm for a logistic regression function provided in an embodiment of the present application;

[0056] Fig. 9 A simulation result diagram of a random optimization algorithm for a logistic regression function provided in an embodiment of the present application;

[0057] Fig.10A simulation result diagram (logarithmic form) of the random optimization algorithm for the logistic regression function provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0059] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0060] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0061] With regard to the technical problems existing in the prior art, the adaptive optimization fast convergence method based on symbolic gradient provided in the present application is intended to solve at least one of the technical problems in the prior art.

[0062] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0063] like Figure 1 As shown, the embodiment of the present application provides a method for fast convergence of adaptive optimization based on signed gradient, which may include the following steps:

[0064] Step 101, for a deterministic optimization scenario, optimizing an objective function according to an adaptive signed gradient descent algorithm;

[0065] Step 102, for the stochastic optimization scenario, optimizing the objective function according to the adaptive symbolic stochastic gradient descent algorithm;

[0066] Among them, both the adaptive symbolic gradient descent algorithm and the adaptive symbolic stochastic gradient descent algorithm use the distance between the current objective function value and the initial objective function value and the preset key hyperparameters to update the step size in each iteration;

[0067] The adaptive signed stochastic gradient descent algorithm allows the random estimate of the gradient to become a biased estimate of the true gradient.

[0068] Optionally, the method of the embodiment of the present application is used to solve the following unconstrained optimization problem:

[0069]

[0070] Among them, the objective function f: is a differentiable function that is allowed to have an inseparable property, that is, the elements of the d dimensions of the parameter x to be optimized may influence each other, and it cannot be guaranteed that each element can independently find the optimal solution without being affected by the other d-1 elements.

[0071] In the following, It means that the objective function f is The gradient at ; sign represents the sign function, which acts on the vector When , the elements in each dimension of the vector are operated one by one; [d] represents the set of positive integers {1,2,…,d}; Represents probability.

[0072] In the embodiments of the present application, an adaptive signed gradient-based optimization algorithm is proposed for the case where the objective function is inseparable, for deterministic optimization scenarios and stochastic optimization scenarios respectively. The algorithm iteration does not depend on the information of the gradient and its norm, and in the stochastic optimization scenario, the random estimate of the gradient is allowed to be a biased estimate of the true gradient, which can be applicable to a large number of practical application scenarios.

[0073] In the embodiment of the present application, for deterministic optimization scenarios, the objective function is optimized according to the adaptive symbolic gradient descent algorithm; for stochastic optimization scenarios, the objective function is optimized according to the adaptive symbolic stochastic gradient descent algorithm. Among them, both the adaptive symbolic gradient descent algorithm and the adaptive symbolic stochastic gradient descent algorithm use the distance between the current objective function value and the initial objective function value and the preset key hyperparameters to update the step size in each iteration. The update of the adaptive step size of the algorithm does not depend on the relevant information of the gradient and its norm, which meets the demand for saving communication costs in distributed application scenarios, avoids transmitting complete gradient information, and instead transmits compressed gradient information (sign function of gradient), effectively reducing the communication overhead between nodes. And the adaptive symbolic stochastic gradient descent algorithm allows the random estimate of the gradient to be a biased estimate of the true gradient, ensuring good convergence performance.

[0074] In an optional embodiment, if Figure 2 As shown, in order to solve the situation where the objective function in the field of deterministic optimization is inseparable, the embodiment of the present application proposes an adaptive sign gradient descent algorithm (Adaptive Sign-based Gradient Descent, referred to as AS-GD):

[0075] Determine initial parameters Initial step size α0∈(0,+∞) and key hyperparameters β,γ∈(0,+∞);

[0076] Initialize iteration index k=0;

[0077] Step 1.1:

[0078] Step 1.2:

[0079] Step 1.3: Let k = k + 1, and return to step 1.1;

[0080] in, It means that the objective function f is The gradient at ; sign represents the sign function.

[0081] The main design idea of ​​the above-mentioned adaptive symbolic gradient descent algorithm is to propose an adaptive step size, using the distance between the current objective function value and the initial objective function value, and introducing two constant key hyperparameters β and γ to update the step size in each iteration.

[0082] In an optional embodiment, the assumptions are determined, and the basic assumptions of the objective function are: strongly convex and smooth, or non-convex and smooth, but satisfying Polyak- Assumption.

[0083] In an optional embodiment, for a deterministic optimization scenario, after optimizing the objective function according to an adaptive signed gradient descent algorithm, the method further includes:

[0084] Based on the basic assumptions of the objective function, the smoothness parameter L and the strong convexity parameter (or Polyak- Assumed parameters)μ;

[0085] Determine the first convergence condition based on key hyperparameters, smoothness parameters, and strong convexity parameters;

[0086] In the present application embodiment, Figure 2 As shown, in order to ensure the convergence of the algorithm, it is necessary to determine the conditions that should be satisfied between the two key hyperparameters β and γ in the algorithm and the parameters L, μ, and f(x0) in the hypothesis, that is, the first convergence condition.

[0087] The convergence of the adaptive signed gradient descent algorithm is demonstrated according to the first convergence condition, and the convergence rate is calculated.

[0088] Finally, we select simulation examples that meet the conditions and reasonably adjust the corresponding key hyperparameters for simulation verification to evaluate the performance of the algorithm. Under reasonable assumptions, the adaptive symbolic gradient descent algorithm proves that the objective function can converge linearly to the neighborhood of the optimal solution, and by properly adjusting the parameters in the algorithm, the neighborhood size can be flexibly reduced or even eliminated.

[0089] From a theoretical perspective, the embodiments of the present application rigorously prove the linear convergence rate of the algorithm under reasonable assumptions, and study the impact of hyperparameter adjustment on the size of the convergence neighborhood, determine the conditions for precise convergence, and meet the needs of performance analysis of the optimization algorithm based on symbolic gradient, that is, to ensure the rapidity and accuracy of the convergence performance during the execution of the algorithm.

[0090] In an optional embodiment, if Figure 2 As shown in the figure, the design and performance analysis of the AS-GD algorithm is extended to the field of stochastic optimization. In order to solve the situation where the objective function in the field of stochastic optimization is inseparable, an adaptive sign-based stochastic gradient descent algorithm (AS-SGD) is proposed:

[0091] Determine initial parameters Initial step size α0∈(0,+∞), key hyperparameters β,γ∈(0,+∞);

[0092] Initialize iteration index k=0;

[0093] Step 2.1: Get a random estimate of the gradient

[0094] Step 2.2:

[0095] Step 2.3:

[0096] Step 2.4: Let k = k + 1, and return to step 2.1;

[0097] in, It means that the objective function f is The gradient at ; sign represents the sign function.

[0098] The adaptive symbolic stochastic gradient descent algorithm also utilizes the difference between the current objective function value and the initial objective function value when designing the adaptive step size, and introduces two key hyperparameters β and γ to update the step size in each iteration.

[0099] In an optional embodiment, if Figure 2 As shown, in the adaptive symbolic stochastic gradient descent algorithm, the SPB assumption is set to allow the random estimate of the gradient to be a biased estimate of the true gradient:

[0100] For any k ≥ 0 and i ∈ [d], the stochastic estimate of the gradient is With the true gradient Satisfy between:

[0101]

[0102] in, and Respectively and The value of the i-th element of .

[0103] In an optional embodiment, if Figure 2 As shown, for the random optimization scenario, after optimizing the objective function according to the adaptive symbolic stochastic gradient descent algorithm, the method further includes:

[0104] Under the premise of satisfying the SPB assumption, define the minimum success probability p min for:

[0105]

[0106] In an optional embodiment, if Figure 2 As shown, under the premise of satisfying the SPB assumption, after defining the minimum success probability, the method further includes:

[0107] Based on the basic assumptions of the objective function, the smoothness parameter L and the strong convexity parameter (or the parameter of the Polyak-Lojasiewicz assumption) μ are determined; where the basic assumptions of the objective function are: strong convexity and smoothness, or non-convexity and smoothness, but satisfying the Polyak-Lojasiewicz assumption. Assumptions;

[0108] The second convergence condition is determined based on key hyperparameters and smoothness parameters, strong convex parameters, and minimum success probability;

[0109] In the embodiment of the present application, in order to ensure the convergence of the adaptive signed stochastic gradient descent algorithm, it is necessary to further determine the two key hyperparameters β and γ in the algorithm and the parameters L, μ, f(x0), p min The condition that needs to be satisfied between is the second convergence condition.

[0110] The convergence of the adaptive symbolic stochastic gradient descent algorithm is demonstrated according to the second convergence condition, and the linear convergence rate can be recalculated in combination with the mathematical expectation.

[0111] Finally, we select simulation examples that meet the conditions and reasonably adjust the corresponding key hyperparameters for simulation verification to evaluate the performance of the algorithm. Under reasonable assumptions, the adaptive symbolic stochastic gradient descent algorithm proves that the objective function can converge linearly to the neighborhood of the optimal solution, and by properly adjusting the parameters in the algorithm, the neighborhood size can be flexibly reduced or even eliminated.

[0112] From a theoretical perspective, the embodiments of the present application rigorously prove the linear convergence rate of the algorithm under reasonable assumptions, and study the impact of hyperparameter adjustment on the size of the convergence neighborhood, determine the conditions for precise convergence, and meet the needs of performance analysis of the optimization algorithm based on symbolic gradient, that is, to ensure the rapidity and accuracy of the convergence performance during the execution of the algorithm.

[0113] In addition, the embodiments of the present application study the impact of the SPB assumption and the minimum success probability on the AS-SGD algorithm, so that the algorithm can still converge effectively when not relying on the strict unbiased estimation assumption, and meet the practical design requirements of stochastic optimization algorithms, that is, the algorithm can still maintain good performance when encountering the situation where there is a large estimation error in the random estimation of the gradient that is common in reality.

[0114] The above-mentioned adaptive signed gradient descent algorithm (AS-GD algorithm) and adaptive signed stochastic gradient descent algorithm (AS-SGD algorithm) are described in detail below with specific embodiments. Both sets of embodiments given are implemented based on the above-mentioned technical solution, and detailed implementation methods and operation processes are given, but the protection scope of this application is not limited to the following embodiments.

[0115] The first group of embodiments is a synthetic data simulation example MOP2 function commonly used in multi-objective optimization scenarios; the second group of embodiments is a logistic regression loss function commonly used in machine learning scenarios. For the objective functions in the two groups of embodiments, the AS-GD algorithm in the deterministic optimization scenario and the AS-SGD algorithm in the stochastic optimization scenario are used for optimization to verify the convergence results of the two types of algorithms, and they are compared with the optimization results of the corresponding objective functions of other classic algorithms in the deterministic optimization and stochastic optimization scenarios to verify the accuracy of the technical solution proposed by the present invention.

[0116] Specifically, in each set of simulation experiments under the deterministic optimization scenario, the AS-GD algorithm is compared and analyzed with the gradient descent algorithm, the signed gradient descent algorithm, the DICHO algorithm, and the Scaled signGD algorithm; in each set of simulation experiments under the stochastic optimization scenario, the AS-SGD algorithm is compared and analyzed with the stochastic gradient descent algorithm, the stochastic signed gradient descent algorithm, the DICHO algorithm, the Scaled signSGD algorithm, and the EF-signSGD algorithm.

[0117] In the legend of the experimental results, the step sizes of the gradient descent algorithm, stochastic gradient descent algorithm, signed gradient descent algorithm, and sign stochastic gradient descent algorithm are all constants, so they are marked as vanilla GD, vanilla signGD, vanilla SGD, and vanilla signSGD in the legend, respectively.

[0118] The experiments involved in the two sets of embodiments were all carried out on a computer equipped with an Intel(R) Core(TM) i7-8750H CPU@2.20 GHz.

[0119] Embodiment 1:

[0120] The experiment is conducted with the MOP2 function as the objective function, which is in the form of:

[0121]

[0122] In the above formula, x i represents the i-th element of the x vector. The optimal value of this objective function is 0, and it is non-convex, smooth, and satisfies the Polyak- Assume that the implementation requirements of the AS-GD algorithm and the AS-SGD algorithm are met.

[0123] In the experiment, d=20 is taken. In order to ensure a fair comparison, the initial parameter x0 of each algorithm is randomly selected as the same value, and parameters that can achieve better convergence effects are selected for each optimization algorithm. Specifically, when setting parameters for the AS-GD algorithm and the AS-SGD algorithm, the key hyperparameters β and γ are set to 0.1 and the initial value of the objective function f(x0), respectively; the gradient descent algorithm, stochastic gradient descent algorithm, symbolic gradient descent algorithm, symbolic stochastic gradient descent algorithm, DICHO algorithm, EF-signSGD algorithm, Scaled signGD algorithm and Scaled signSGD algorithm all use a step size of 0.1. The number of iterations k is used as the horizontal axis, and the objective function value f(x0) at the kth iteration is used as the horizontal axis. k ) is the vertical axis, and the simulation experiment under the deterministic optimization scenario for the MOP2 function is shown in Figure 3 In addition, to test the linear convergence of the algorithm, the number of iterations k is used as the horizontal axis and the objective function value f( x k) and the optimal value f * The logarithmic scale of the gap between is the ordinate, and the corresponding logarithmic simulation experimental results are as follows Figure 4 shown.

[0124] Similarly, the experiments on the random optimization scenario of the MOP2 function are carried out. The experimental results in general form and logarithmic form are as follows: Figure 5 and Figure 6 shown.

[0125] Embodiment 2:

[0126] The experiment is conducted using the common loss function logistic regression function in the field of machine learning, which is in the form of:

[0127]

[0128] In the above formula, represents the number of samples in the dataset, represents the i-th data sample, b i ∈{-1,1} represents a i The l2-regularization term is introduced to ensure that the objective function is strongly convex. Therefore, the objective function is strongly convex and smooth, which meets the requirements of the embodiment.

[0129] In the experiment, the Ionosphere dataset, which is commonly used in logistic regression experiments, was used, with n=351 and d=34. To ensure a fair comparison, the initial parameter x0 of each algorithm was randomly selected to be the same value, and each optimization algorithm was selected to achieve a better convergence effect. Specifically, when setting parameters for the AS-GD algorithm and the AS-SGD algorithm, β was set to 0.05, and the optimal value of the objective function was given a pre-estimated value f using the Python scipy.optimize library. b Under the premise of setting γ as the initial value of the objective function f(x0) and f b The difference is ; the gradient descent algorithm, stochastic gradient descent algorithm, signed gradient descent algorithm, signed stochastic gradient descent algorithm, DICHO algorithm, EF-signSGD, Scaled signGD algorithm and Scaled signSGD algorithm all use a step size of 0.05.

[0130] Similarly, the number of iterations k is used as the horizontal axis, and the objective function value f(x k ) is the vertical axis, and the simulation experiment under the deterministic optimization scenario is carried out for the logistic regression function using the Ionosphere data set. Figure 7 In addition, to verify the linear convergence of the algorithm, the number of iterations k is used as the horizontal axis and the objective function value f(x k ) and the estimated value f of the optimal value of the objective function b The logarithmic scale of the gap between is the ordinate, and the corresponding logarithmic simulation experimental results are given as follows Figure 8 shown.

[0131] Similarly, for the logistic regression function using the Ionosphere dataset, the experimental results in the random optimization scenario are shown in the general form and the logarithmic form. Fig. 9 and Fig.10 shown.

[0132] The simulation experimental results of the above two groups of embodiments show that both the signed gradient descent and the signed stochastic gradient descent algorithms with a constant step size cannot converge, but oscillate near the optimal solution. Whether in a deterministic optimization scenario or a stochastic optimization scenario, the DICHO algorithm can converge, but cannot converge to the optimal value of the objective function. According to the logarithmic experimental result graph, AS-GD, AS-SGD, gradient descent, stochastic gradient descent, Scaled signGD, Scaled signSGD and EF-signSGD algorithms all show linear convergence, and the linear convergence rates of the AS-GD algorithm and the AS-SGD algorithm are comparable to the convergence rates of other methods that can achieve linear convergence in their respective experiments in the same group.

[0133] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A fast convergence method for adaptive optimization based on signed gradient, characterized in that: The method comprises: For deterministic optimization scenarios, the objective function is optimized according to the adaptive symbolic gradient descent algorithm; For random optimization scenarios, the objective function is optimized according to the adaptive symbolic stochastic gradient descent algorithm; Wherein, the adaptive signed gradient descent algorithm and the adaptive signed stochastic gradient descent algorithm both use the distance between the current objective function value and the initial objective function value and preset key hyperparameters to update the step size in each iteration; The adaptive signed stochastic gradient descent algorithm allows the random estimate of the gradient to be a biased estimate of the true gradient.

2. The method for rapid convergence of adaptive optimization based on signed gradient according to claim 1, characterized in that: The adaptive signed gradient descent algorithm is: Determine initial parameters Initial step size α0∈(0,+∞) and key hyperparameters β,γ∈(0,+∞); Initialize iteration index k=0; Step 1.1: Step 1.2: Step 1.3: Let k = k + 1, and return to step 1.1; in, It means that the objective function f is The gradient at ; sign represents the sign function.

3. The method for rapid convergence of adaptive optimization based on signed gradient according to claim 2, characterized in that: The basic assumptions of the objective function are: strongly convex and smooth, or non-convex and smooth, but satisfying Assumption.

4. The method for rapid convergence of adaptive optimization based on signed gradient according to claim 3, characterized in that: For the deterministic optimization scenario, after optimizing the objective function according to the adaptive signed gradient descent algorithm, the method further includes: Determine smoothness parameters and strong convexity parameters based on the basic assumptions of the objective function; Determine a first convergence condition based on the key hyperparameter, the smoothness parameter, and the strong convexity parameter; The convergence of the adaptive signed gradient descent algorithm is demonstrated based on the first convergence condition.

5. The method for rapid convergence of adaptive optimization based on signed gradient according to claim 1, characterized in that: The adaptive signed stochastic gradient descent algorithm is: Determine initial parameters Initial step size α0∈(0,+∞), key hyperparameters β,γ∈(0,+∞); Initialize iteration index k=0; Step 2.1: Get a random estimate of the gradient Step 2.2: Step 2.3: Step 2.4: Let k = k + 1, and return to step 2.1; in, It means that the objective function f is The gradient at ; sign represents the sign function.

6. The method for rapid convergence of adaptive optimization based on signed gradient according to claim 5, characterized in that: In the adaptive signed stochastic gradient descent algorithm, the SPB assumption is set to allow the random estimate of the gradient to be a biased estimate of the true gradient: For any k ≥ 0 and i ∈ [d], the stochastic estimate of the gradient is With the true gradient Satisfy between: in, and Respectively and The value of the i-th element of .

7. The method for rapid convergence of adaptive optimization based on signed gradient according to claim 6, characterized in that: After optimizing the objective function according to the adaptive symbolic stochastic gradient descent algorithm for the stochastic optimization scenario, the method further includes: Under the premise of satisfying the SPB assumption, define the minimum success probability p min for:

8. The method for rapid convergence of adaptive optimization based on signed gradient according to claim 7, characterized in that: After defining the minimum success probability under the premise of satisfying the SPB assumption, the method further includes: Determine smoothness parameters and strong convexity parameters based on the basic assumptions of the objective function; Determining a second convergence condition based on the key hyperparameter, the smoothness parameter, the strong convexity parameter, and the minimum success probability; Demonstrating the convergence of the adaptive signed stochastic gradient descent algorithm according to the second convergence condition; The basic assumptions of the objective function are: strongly convex and smooth, or non-convex and smooth, but satisfying Assumption.

Images