A zeroth order optimizer based on variance reduction techniques

Abstract

The application designs a natural evolution strategy optimization method based on variance reduction technology, aiming at solving the problem of large gradient estimation variance and slow convergence speed in traditional natural evolution strategy. The method combines variance reduction technology, introduces global average gradient and small batch gradient correction in the optimization process, significantly reduces the variance of gradient estimation, improves the optimization accuracy and efficiency. Specifically, the method includes the following steps: first, initialize the optimization parameters, including the initial solution, learning rate and step size; then calculate the global gradient by input data, which is used for subsequent correction; in the inner loop, calculate the objective function value and gradient for small batch perturbation samples, and combine the global gradient for variance correction to update the optimization parameters. Compared with the traditional method, the application has the advantages of high gradient estimation accuracy and fast convergence speed, and is suitable for black box objective function optimization scenarios, and has wide application value in the field of machine learning model training.

Description

Technical Field

[0001] This invention belongs to the field of machine learning, specifically relating to a zero-order optimization system and method based on variance reduction technology, which is mainly used to solve black-box optimization problems where the objective function is not differentiable or the gradient is unavailable. Background Technology

[0002] Optimization problems involve finding the optimal solution to an objective function given constraints. Zero-order optimization is a method for solving optimization problems, characterized by its reliance on the objective function value alone, without requiring gradient information. Unlike first- or second-order optimization methods, zero-order optimization does not require the objective function to be differentiable, thus it can be used to handle situations where the objective function is not explicitly available but can be evaluated in a black-box manner. Zero-order optimization methods are particularly effective for certain objective functions, especially those that are discontinuous, non-smooth, or whose gradients are difficult to compute. For example, if the objective function in a black-box optimization problem is unknown, its gradient will be difficult to compute, making zero-order optimization methods more suitable than optimization methods that require calculating the objective function's gradient.

[0003] In zero-order optimization methods that use finite difference methods to replace gradient calculation, it is usually necessary to collect samples and calculate the objective function for the corresponding sample points to estimate the gradient, which leads to noise. This is because in finite difference methods, gradient estimation depends on sampling the objective function values, and these samples themselves may have uncertainties. Furthermore, in addition to the collected samples, the input values ​​used in calculating the objective function values ​​also introduce uncertainty or noise into the gradient estimation. Typically, the input is considered accurate, and the noise comes from the model itself and the output, while noise from the input is ignored. Since the input can be considered as sampled from a random distribution, the input, as a random variable, also introduces noise into the optimization process. Addressing this noise is of high application value for optimization methods. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a zero-order optimization system and method based on variance reduction technology, which can handle noise introduced by the input during the optimization process.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A zero-order optimization system based on variance reduction technology is characterized by comprising the following two modules:

[0007] The Natural Evolutionary Strategy (NES) is the core optimization framework of this invention. It primarily searches for the optimal solution on a complex objective function by simulating an evolutionary process. The basic principle of NES is to optimize the expected value of the objective function based on a parameterized distribution, where is a probability distribution controlled by policy parameters, and is the objective function value.

[0008] This invention introduces a variance reduction module into the NES framework to reduce the variance caused by input randomness during the sampling of objective function values, thereby improving the accuracy of gradient estimation and optimization efficiency.

[0009] Furthermore, this invention also proposes a zero-order optimization implementation method based on variance reduction technology as described above, characterized by the following steps:

[0010] Input: learning rate, initial step size, initial solution.

[0011] Outer loop:

[0012] 1) Sampling perturbation: A batch of Gaussian random vectors is sampled from the standard normal distribution, with a sampling size of B;

[0013] 2) Sample input vector: Sample input sample vectors from the sample set, with a size of S;

[0014] 3) Calculate the objective function value and gradient: Calculate the overall objective function value using the current solution with added perturbation parameters and the input samples, and calculate the overall estimated gradient using the objective function value of the current solution;

[0015] 4) Initialize parameters: Record the current solution and set the initial value of the inner loop to the current solution;

[0016] 5) Start the inner loop.

[0017] Inner loop:

[0018] 1) Mini-batch sampling perturbation: Sample a mini-batch of Gaussian random vectors from the standard normal distribution, with a sampling size of n;

[0019] 2) Sampling input vector: Randomly select an input sample from the input sample vector sampled from the outer layer as the inner layer input sample;

[0020] 3) Calculate the mini-batch objective function value and gradient: Calculate the objective function value using the current solution of the inner loop with added mini-batch perturbation parameters and the input samples, and calculate the mini-batch estimated gradient using the objective function value of the current solution of the inner loop;

[0021] 4) Calculate the relevant gradient: Use the current solution and input samples recorded by the outer loop with added mini-batch perturbation parameters to calculate the objective function value, and use the objective function value of the current solution recorded by the outer loop to calculate the relevant gradient;

[0022] 5) Calculate the corrected gradient: Use the mini-batch estimated gradient - the relevant gradient + the overall estimated gradient as the corrected gradient;

[0023] 6) Parameter update: Update the parameters according to the gradient, and return to the outer loop after M updates.

[0024] To optimize the above technical solution, the specific measures also include:

[0025] 1) Adaptive adjustment of learning rate α: Initially set to 0.01. A dynamic learning rate is used, adjusting the learning rate based on gradient changes to avoid oscillations caused by excessively large updates or slow convergence caused by excessively small updates;

[0026] 2) Dynamic adjustment of step size σ: Initially set to 0.01. It is dynamically adjusted based on the gradient variance during the optimization process. If the gradient estimation variance is large, the step size is increased for better exploration; if the variance is small, the step size is decreased to improve accuracy.

[0027] 3) Selection of mini-batch size n: The value of n is set to 32 or 64 according to the problem size and computing resources, taking into account both computational efficiency and gradient estimation accuracy;

[0028] 4) Selection of the number of iterations M: The parameter M is the number of iterations in the inner loop. In each round, with a fixed noise intensity and random perturbation vector, it represents the number of gradient updates performed to optimize the current parameters. This is a core parameter in the algorithm implementation, directly affecting the accuracy of the optimization results and the algorithm's running efficiency. Increasing M can increase the number of updates that can be performed in one full gradient calculation; however, correspondingly, the difference between the current solution and the solution stored in the outer loop also increases with the number of updates, thus introducing a certain bias. Depending on the specific optimization problem, it can be set to 5 or 10.

[0029] The beneficial effects of this invention are: it designs an optimization system with a natural evolution strategy (NES) featuring variance reduction, aiming to improve convergence efficiency and result stability by reducing the variance of gradient estimation. Variance reduction technology makes parameter updates more stable and reliable by reducing the fluctuation amplitude of gradient estimation. Compared to traditional methods, this technique can find the optimal solution of the objective function more quickly. Attached Figure Description

[0030] Figure 1 This is a framework diagram of a zero-order optimization system based on variance reduction techniques. Detailed Implementation

[0031] The invention will now be described in further detail with reference to the accompanying drawings.

[0032] like Figure 1 The main idea of ​​the zero-order optimization system based on variance reduction technology shown is to divide the system into two modules: a natural evolution strategy and variance reduction. The specific implementation involves the following steps, corresponding to the core algorithm flow and module design:

[0033] 1) Parameter initialization: Initialize the optimization parameters and set the initial solution vector. Learning rate Initial step size Number of inner iterations ;

[0034] 2) Random sampling: Generate samples that follow a Gaussian distribution. random perturbation and random samples in the input sample set ;

[0035] 3) Calculate the global average gradient: for each random perturbation Calculate the objective function value Estimate global gradient This average gradient is used to correct the variance of subsequent gradients;

[0036] 4) Inner layer iterative optimization: For each round of outer layer iteration, record the current solution. Execute the inner loop Next, optimization is completed through mini-batch perturbation and gradient correction. The specific steps are as follows:

[0037] Mini-batch perturbation sampling: from Gaussian distribution Randomly sample a small batch perturbation ;

[0038] Objective function calculation: for the current parameters Use each random perturbation Calculate the objective function value and ;

[0039] Calculate the mini-batch gradient: Estimate the mini-batch gradient ;

[0040] Calculate the relevant gradient: Use the solution from the outer record to calculate the relevant gradient. ;

[0041] Variance reduction correction: Using historical gradient information and the current gradient estimate, variance is corrected to improve accuracy. ;

[0042] Parameter update: using the corrected gradient Update and optimize parameters If the inner loop has already been executed Then, return to the outer loop.

[0043] 5) The outer layer reaches the termination condition, and the optimization is complete.

Claims

1. A natural evolution strategy optimization method based on variance reduction technology, characterized in that, The method includes the following steps: 1) Initialize the optimization parameters, including the initial solution vector, learning rate, step size, and number of inner iterations; 2) Generate Gaussian distributed perturbation samples, sample input samples, and calculate the global average gradient; 3) In each round of inner-layer optimization, calculate the mini-batch gradient and correlation gradient for the mini-batch perturbation samples, and use the global average gradient and correlation gradient to perform variance reduction correction on the mini-batch gradient and update the optimization parameters; 4) Dynamically adjust the step size according to the optimization process; 5) Repeat the outer iteration until the convergence condition is met or the maximum number of iterations is reached.

2. The optimization method according to claim 1, characterized in that, The formula for calculating the global average gradient is: , where F(·) is the objective function and ξ is the input data.

3. The optimization method according to claim 1 or 2, characterized in that, The formula for calculating the mini-batch gradient is:

4. The optimization method according to claim 1 or 2, characterized in that, The formula for calculating the relevant gradient is:

5. The optimization method according to claim 1 or 2, characterized in that, The formula for calculating the correction gradient is:

6. The optimization method according to claim 1, characterized in that, The optimized parameter update formula is as follows: , where α is the learning rate.

7. The optimization method according to claim 1, characterized in that, The initial step size is set to 0.01, the learning rate is set to 0.01, and the outer iteration stops when the gradient magnitude is lower than a preset threshold or the number of outer iterations reaches the maximum value.

8. The optimization method according to claim 1, characterized in that, This method is applicable to scenarios such as machine learning model parameter optimization and black-box function optimization.

Images