Gradient estimation operator, method and device for large model training

Abstract

The invention discloses a gradient estimation operator, method and device for large model training, relates to the technical field of reinforcement learning, and solves the technical problems that an existing gradient estimation method is relatively high in calculation overhead and memory overhead, and low variance and high-order differentiability are difficult to consider at the same time. The method comprises the steps that a control variable acts on an original function, the result of a random calculation graph is indirectly changed, and gradient mapping of a loss function # imgabs0 # is converted into # imgabs2 # from # imgabs1 #. According to the method, control variable design is transferred to the original function layer from a gradient space, explicit calculation and storage of an intermediate control variable function are eliminated, and therefore the control variable design is optimized. And the calculation and memory overhead is reduced fundamentally.

Description

Technical Field

[0001] The present invention relates to the field of optimization computing technology, and in particular to a gradient estimation operator, method and device for large model training. Background Art

[0002] Existing gradient estimation methods mainly include:

[0003] (1) REINFORCE and its Baseline technology estimate the gradient by replacing the loss function with the log-likelihood derivative of the expected value. Although this method is suitable for discrete scenarios, it has obvious limitations in solving high variance problems.

[0004] (2) Leave-One-Out control variable technique constructs random control variables by using the leave-one-out method on the samples, aiming to reduce the noise in the gradient estimation process;

[0005] (3) REBAR and RELAX methods introduce continuous relaxation techniques to construct control variables for discrete variable problems, but each sample often requires multiple evaluations;

[0006] (4) RODEO and other advanced control variable designs customize exclusive control variables for each sample by introducing Stein operators or combining Markov process information, further reducing variance and improving estimation stability, but the implementation complexity is relatively high;

[0007] (5) Historical gradient memory methods such as SAGA and SVRG use historical gradient information to reduce estimation errors through periodic corrections, but they may face memory overhead and information storage inconvenience in large-scale data and high-dimensional parameter spaces;

[0008] Existing gradient estimation techniques face numerous challenges and limitations. First, methods such as REBAR and RELAX typically require multiple function or gradient evaluations (usually three times) for each sample, significantly increasing computational overhead for large-scale models or datasets. Second, traditional control variable methods often require explicit calculation and storage of intermediate variables associated with each sample and all model parameters. This leads to additional memory overhead and reduced efficiency in practical applications when the parameter scale is large or the number of samples is large. Third, some methods, such as reparameterization techniques, rely on the assumption of continuous probability distributions and lack universality for discrete scenarios. Advanced control variable designs, such as RODEO, while offering significant theoretical advantages, are complex in structure and difficult to implement. They often require separate control variables for each sample, increasing the burden of code implementation and parameter debugging. Finally, when extending traditional methods to gradient estimation of arbitrary-order derivatives, it is difficult to simultaneously achieve both low variance and high-order differentiability, or additional computational costs may be incurred, limiting their potential for application in complex optimization problems. Summary of the Invention

[0009] In order to solve the problems existing in the above-mentioned prior art, the present invention provides a gradient estimation operator, method and device for large model training, which solves the technical problems that the existing gradient estimation method has large computational overhead and memory overhead, and is difficult to simultaneously take into account low variance and high-order differentiability.

[0010] A gradient estimation operator for large model training, including: applying control variables to the original function, indirectly changing the result of the random calculation graph, and converting the loss function The gradient map from Convert to It is expressed as follows:

[0011]

[0012] Use Monte Carlo sampling and K samples to estimate the gradient:

[0013]

[0014] f η (x) represents the proxy loss function, h(x) represents the control variable function corresponding to each sample x, a is a real number in (0,1], is a dynamically adjusted baseline or control variable function, for The expected form of θ depends on θ and has the same gradient expectation with respect to θ. exp(·) is the exponential function, ⊙ refers to the Magic-Box operator, represents the expected return, x k is the input sample, α is a scalar hyperparameter, p θ (x) is the probability distribution, is the gradient calculation, is the loss function, is the gradient estimation operator, and K is the number of samples.

[0015] Furthermore, in a near-optimal design:

[0016]

[0017] Where, It is a low-order linear approximation of f(x), obtained by linear regression fitting.

[0018] Furthermore, in RLOO:

[0019]

[0020] Furthermore, in Double CV:

[0021]

[0022] Where b(x) is f η First-order Taylor approximation of (x)

[0023] Furthermore, in the neural network model parameterized by the parameter φ:

[0024]

[0025] Where h φ (x,f η (x)) represents a function from a family of flexible parameters with parameter φ.

[0026] A gradient estimation method for large model training using CVor optimization include:

[0027] Step 1: Input the target function f η (x), distribution p θ (x), aggregate function and Initialize θ, η, α = 0;

[0028] Step 2: For i=1,2,…, from p θ (x) extract K samples {x i} i∈{1,…,K} , that is, x i ~p θ (x);

[0029] Step 3: Calculation

[0030] Step 4: Find the gradient of CVor loss:

[0031]

[0032] Step 5: Calculate CVor gradient

[0033]

[0034] Step 6: Update the objective function parameters θ, learning rate η and hyperparameter α.

[0035] Furthermore, update the parameter θ which is equivalent to the objective function:

[0036] Furthermore, update the learning rate η:

[0037] Furthermore, update the hyperparameter α:

[0038] A gradient estimation device for large model training, characterized in that it is used to implement a gradient estimation method for large model training, including a control unit, an instruction cache module, a direct memory access module, a data processing module and a data cache module,

[0039] The control unit is used to call instructions from the instruction cache module to control the data processing module;

[0040] The instruction cache module is used to cache sampling instructions, differential instructions and parameter update instructions in response to instruction calls from the control unit;

[0041] The data processing module includes a sampling calculation unit, a gradient estimation unit, and a parameter updating unit. The sampling calculation unit is used to perform sampling according to a sampling instruction and calculate the objective function and H(x) according to the sampled samples. The gradient estimation unit is used to perform Cvor gradient estimation and parameter optimization calculation according to a differential instruction. The parameter updating unit is used to update the model parameters to be optimized according to the parameter update instruction and the Cvor gradient calculation.

[0042] The data cache module includes a data cache unit, a gradient cache unit and a parameter cache unit. The data cache unit is used to cache sampling samples and expected estimation results of the sampling samples, the gradient cache unit is used to cache gradient calculation results, and the parameter cache unit is used to cache updated parameters.

[0043] The beneficial effects of the present invention include:

[0044] This paper adopts a new CVor control variable operator. By transferring the control variable design from the gradient space to the primitive function level, it eliminates the explicit calculation and storage of the intermediate control variable function, fundamentally reducing the computational and memory overhead. Compared with traditional methods, CVor can achieve high-order differentiability of arbitrary-order derivatives while ensuring unbiased and low-variance gradient estimation, effectively avoiding repeated evaluations and complex sample-specific design issues. In addition, this method simplifies code implementation and parameter debugging, significantly improving the practicality and stability in large-scale models and discrete scenarios, and overall improving the efficiency and performance of deep learning training. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a random computation graph of a gradient estimation operator CVor for large model training involved in an embodiment of the present application.

[0046] Figure 2 The embodiments of this application involve the use of CVor in VAE experiments on four datasets, compared with the most advanced solutions, for the gradient estimator variance.

[0047] Figure 3 The variance of the gradient estimator using three different control variables (CV) of CVor in the RL experiments in six environments involved in the embodiments of this application is compared with the baseline.

[0048] Figure 4 The ELBO evaluation results of three different control variables (CV) of CVor in VAE experiments are compared with the state-of-the-art solutions in four data sets involved in the embodiments of this application.

[0049] Figure 5 The embodiments of this application involve evaluating the reward results of three different forms in RL experiments in six environments compared with the baseline.

[0050] Figure 6 This section compares the memory usage of different methods under different batch sizes involved in the embodiments of this application.

[0051] Figure 7 This is a comparison of the time complexity of different methods involved in the embodiments of this application under different batch sizes.

[0052] Figure 8 This is the change in the use of CVor in different scenarios involved in the embodiments of this application.

[0053] Figure 9 This is a schematic diagram of the operation of a gradient estimation device for large model training involved in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0055] Computing the exact gradient of the objective function on large-scale data is infeasible due to the extremely high computational and storage overhead. In this context, gradient estimation based on sampled data becomes essential for efficiently updating model parameters. A common approach is to randomly select small batches of data (stochastic gradient descent, SGD). However, due to the high variance of the estimated gradient, it can easily deviate from the optimal update direction. Therefore, developing strategies to reduce the variance of the gradient has become an important goal in gradient estimation.

[0056] To improve gradient estimation, a typical method is the reparameterization technique, which estimates the gradient through a function g(θ,∈), where θ represents the model parameters and ε is noise with a specific distribution. This method reduces the variance by attributing randomness to the noise component ε, but it only works for continuous probability distributions and is therefore not applicable to discrete scenarios.

[0057] Another common strategy is the REINFORCE estimator, which estimates the gradient by taking an alternative loss function of the log-likelihood derivative of the expected value. This approach is suitable for discrete scenarios, but does not perform well in solving high-variance problems.

[0058] To overcome the shortcomings of the above methods, control variables (CVs) are proposed to estimate the gradient in discrete scenarios using the following formula:

[0059]

[0060] Among them, α is a scalar hyperparameter, g θ (x) represents the original gradient, c θ (x) is the control variable, z θ (x) represents the estimated gradient. Basic theory shows that when c θ (x) and g θ (x) is strongly correlated, z θ (x) can provide low-variance gradient estimates [7] Based on this principle, current research focuses on designing efficient c θ (x).

[0061] While existing control variable methods offer some benefits, they also raise the question of how to deploy them efficiently, especially in learning models with large parameter sizes.

[0062] The control variables of existing methods require explicit calculation and storage of intermediate variables c θ (x), the gradient g of this variable with each sample x and all parameters θ θ (x) related.

[0063] Furthermore, complex operators also introduce additional computational overhead to the training process. Especially when faced with large-scale models, this computational and storage requirement can be quite substantial, and may even offset the benefits of gradient estimation based on sampled data.

[0064] In this context, this embodiment proposes a gradient estimation operator for large model training, called CVor. CVor eliminates the need to explicitly calculate the intermediate variable c θ (x), so that no additional computation and storage costs are required when applying control variables.

[0065] Unlike traditional control variable methods that focus on designing the gradient g θ (x)c θ (x) Different, CVor focuses on design c θ The primitive function of (x) Through based on The function acts on the objective function

[0066] like Figure 1 As shown, CVor changes the random computation graph of gradient computation and maps the gradient from Transfer to control variable structure The specific expression is as follows:

[0067]

[0068] Where, f η (x) represents the proxy loss function, h(x) represents the control variable function corresponding to each sample x, a is a real number in (0,1], is a dynamically adjusted baseline or control variable function, for The expected form of θ depends on θ and has the same gradient expectation with respect to θ. exp(·) is the exponential function, ⊙ refers to the Magic-Box operator, represents the expected return, x k is the input sample, α is a scalar hyperparameter, p θ (x) is the probability distribution, is the gradient calculation, is the loss function, is the gradient estimation operator, and K is the number of samples.

[0069] The Magic-Box operator expression is:

[0070]

[0071] Compared with the traditional direct design control variable c θ (x) and acts on the gradient. CVor focuses on modifying the function ,in This approach simplifies the implementation of control variables and eliminates the need to explicitly compute and store the θ-specific c θ (x) without introducing additional computational and memory overhead, which is especially significant in large-scale models. It is theoretically proven that Cvor is unbiased and differentiable at any order.

[0072] Traditional C-based θ (x) method first calculates the parameter gradient c θ(x), and then use these gradients to calculate new gradients to update the model. CVor skips the first step of gradient calculation and directly uses x and To deduce the sample Recalculate CVor loss And update the model through automatic differentiation. Although the automatic differentiation process will calculate and The gradient of , but it should be emphasized that automatic differentiation is efficient and the gradient is released immediately after use. By eliminating one step of gradient calculation, CVor reduces both processing time and memory cost compared to traditional methods. For example, Designed for The amount of storage required is proportional to the number of x, and Because operation, whose resource requirements are proportional to the number of parameters θ corresponding to each x.

[0073] In another embodiment, a The approximate optimal design of

[0074]

[0075] Where, is a low-order linear approximation of f(x), which can be obtained by sampling a set of ({x},{f(x)}) pairs and fitting them using linear regression. In the case of this value function baseline, It is desirable to design for bias elimination during the automatic differentiation process.

[0076] In another embodiment, REINFORCE leave-one-out (RLOO) is a method that uses f η (x) information to construct a representative method of the baseline. The RLOO estimator requires the sample size K ≥ 2 and uses the leave-one-out operator to generate the baseline b(x). Its standard form is as follows:

[0077]

[0078] Among them, the sample set is recorded as x=(x1,x2,...,x n ),and is the leave-one-out baseline. Although a correction term is usually required to ensure unbiasedness, in some cases it can be omitted because its overall expected value is zero.

[0079]

[0080] Similar to the REINFORCE estimator with baseline, the RLOO estimator is unbiased and has η(x) and b x\k (x) have sufficient correlation between them, thus showing high efficiency in reducing variance.

[0081] Based on formula (3.1), this embodiment is defined in RLOO form:

[0082]

[0083] In this case, the gradient estimation based on CVor is achieved by leaving one out, that is, eliminating one sample and calculating the function f(x k ) and the mean value of the remaining samples h(x k )=c x\k (x) is used as the estimated gradient; when the function and When interacting, the gradient of the logarithm of the model probability distribution is generated

[0084] Under this leave-one-out baseline setting,

[0085]

[0086] It plays a role in the automatic differentiation process and helps reduce variance without introducing bias.

[0087] In another embodiment, the Double-CV version of the CVor function h(x) can be expressed as:

[0088]

[0089] where b(x) is f η (x) is a first-order Taylor approximation. In this case,

[0090]

[0091] In another embodiment, a more general The form is to use a neural network model parameterized by the parameter φ. According to formula (1.8), h φ (x) and f η The correlation between (x) plays an important role in reducing the variance. Therefore, the goal is to enhance h φ (x) and f η (x). It should be noted that h φ (x) can be obtained from f through knowledge distillation or model quantization η (x) is obtained.

[0092] A neural network-based free-form design is provided for h φ (x), consider the following form

[0093]

[0094] where h φ (x,f η (x)) represents a function from a family of flexible parameters with parameter φ. This function is chosen due to its significantly lower computational cost. Importantly, the design of this surrogate function does not require additional computational evaluation beyond the computational cost. Required assessment.

[0095] In addition, the network parameters φ can be optimized in real time to minimize the variance of the estimator, which is similar to the method for updating α in Algorithm 1.

[0096] Although this surrogate function design does not require additional evaluation, the optimization of the parameter φ may introduce computational overhead, which depends on the complexity of the network and the selected optimization method. When implementing this method, a trade-off needs to be made between computational efficiency and variance reduction. Finally, the flexible design of The performance depends on the specific task or problem.

[0097] In another embodiment, a gradient estimation method for large model training is provided, using CVor optimization include:

[0098] Step 1: Input the target function f η (x), distribution p θ (x), aggregate function and Initialize θ, η, α = 0;

[0099] Step 2: For i=1,2,…, from p θ (x) extract K samples {x i} i∈{1,…,K} , that is, x i ~p θ (x);

[0100] Step 3: Calculation

[0101]

[0102] Step 4: Find the gradient of CVor loss:

[0103]

[0104] Step 5: Calculate CVor gradient

[0105]

[0106] Indicates that after CVor conversion, and is the objective function gradient estimator of the parameters. For simplicity, we use express It can be seen that the function The structural form of is consistent with the form of CVs that inherits the advantages of control variables; fundamentally, CVor acts as an operator that converts the objective function constructed by the automatic differentiation framework (such as .detach() or .stop_gradient()) The computational graph is converted into a control variable structure

[0107] Step 6: Update the objective function parameters θ, learning rate η and hyperparameter α.

[0108] Update the parameters θ which is equivalent to the objective function:

[0109]

[0110] Update the learning rate η:

[0111]

[0112] Update the hyperparameter α:

[0113]

[0114] Specifically, in reinforcement learning, the CVor operator can be used to improve gradient estimation in the policy gradient method, thereby achieving more stable and efficient policy updates. The following uses a policy gradient-based reinforcement learning task as an example to illustrate how the technical solutions in the above algorithm can be applied to reinforcement learning:

[0115] (1) Task description

[0116] For example, suppose we are training an agent to perform a task in a discrete action space, whose policy is determined by the parameters θ, i.e., p θ (x) represents the probability distribution of selecting action x in state. The goal is to maximize the cumulative reward, and the corresponding optimization objective can be expressed as where f η (x) can be viewed as an evaluation of the quality of a strategy using a state-action value function or advantage function (represented by the parameter η).

[0117] (2) Sampling and objective function construction

[0118] In each training cycle, the agent uses the current strategy p θ(x) Sample K trajectories or samples. For each trajectory, the reward and state value are calculated using the environmental feedback to construct the objective function That is, the expected return of the strategy under the current state.

[0119] (3) Gradient estimation using CVor operator

[0120] The traditional policy gradient method is prone to large variance of gradient estimation due to sample fluctuations, while the CVor operator introduces two set functions and (It can be regarded as a dynamically adjusted baseline or control variable function) Construct a new gradient map and decompose the original gradient into a low-variance part and a control variable part, thereby significantly reducing the estimated variance. The specific process is as follows:

[0121] Calculate the objective function

[0122] ·pass Calculate the gradient and obtain the CVor gradient Directly use this low-variance gradient to update the policy parameters θ.

[0123] Parameter update and optimization: In addition to updating the policy parameters θ, the algorithm also performs gradient updates on the evaluation function parameters η (e.g., the value network or advantage network) and adjusts the hyperparameter α (which controls the amount of variation) based on the squared norm of the gradient to ensure the stability of the overall gradient estimate.

[0124] (4) Practical advantages

[0125] By introducing the CVor operator into the policy gradient, the reinforcement learning agent can obtain gradient estimates with lower variance on the sampled data, making it more stable when updating the policy and reducing fluctuations caused by gradient noise. At the same time, since it eliminates the dependence on explicit calculation and storage of intermediate variables, this method can significantly reduce computational and memory overhead in large-scale environments or complex tasks, making the CVor operator easily integrated into the existing deep reinforcement learning code framework and suitable for large-scale training tasks.

[0126] In another embodiment, the efficiency of CVor is analyzed on four VAE tasks and six RL benchmarks. In the RL part, different simulators such as classic-control, Box2D and MuJoCo are used to test the method. Three forms of Settings: Near-optimal structure, RLOO, and neural network-based control variables.

[0127] 1. Experimental Setup

[0128] All experiments were conducted on a high-performance workstation (i9-12900K, 64G memory, RTX3090). The detailed settings of the baseline and the three forms of the CVor-based control variable method are as follows:

[0129] The value function baseline is constructed using the standard BASELINE. It represents a special type of control variable, in which As a construct.

[0130] Approximate Optimal Architecture (AO) is a practical approximation to the optimal form by using a low-order linear approximation To simplify the calculation of the complex function f(x), the function structure obtained is

[0131] RLOO is a classic design known for its simplicity and effectiveness, and has been used in many schemes including Double-CV and RODEO. Based on this method, the function Defined as ∑ x c x\k (x)log p θ (x).

[0132] The neural network (hereinafter referred to as NN) represents a single-layer neural network design, which contains two fully connected layers with 128 neurons each. The outputs of these two layers are activated by the LeakyReLU function and then input into the final fully connected layer.

[0133] The experiments are divided into two parts: the first part focuses on training VAEs using Bernoulli latent variables; the second part involves RL experiments on six benchmarks.

[0134] 2. VAE Experiment

[0135] The VAE experiment uses the joint density p θ (y,x), where x is the latent variable and θ represents the model parameters. The main training goal is to minimize the evidence lower bound (ELBO), which involves an auxiliary inference network q η (x|y) and defined as f(x) = log p θ (y,x)-log q η Function of (x|y).

[0136] The experiment is conducted on p under high dimensional distribution θ (x) and q η (x|y) is used for training, and the random variables in each dimension are independently Bernoulli distributed.

[0137] VAE dataset and experimental settings:

[0138] VAEs are tested using real-world datasets, using the same training / testing split as in previous work. These datasets include StaticMNIST, DynamicMNIST, FashionMNIST, and ContinuousFashion.

[0139] The details of the four datasets are as follows:

[0140] StaticMNIST: This dataset is derived from the original MNIST dataset and uses a fixed threshold binarization technique. Pixel values ​​greater than 0.5 are set to 1, and those less than or equal to 0.5 are set to 0, forming a static binary image set.

[0141] DynamicMNIST: Also derived from the original MNIST dataset, but using dynamic Bernoulli binarization. Pixel values ​​are binarized in a random manner based on their original grayscale intensities, resulting in a dynamically changing binary image dataset.

[0142] FashionMNIST: This is a modern dataset proposed by Zalando as an alternative to the traditional MNIST dataset. It contains 70,000 28x28 pixel grayscale images classified into 10 clothing and footwear categories. 60,000 images are used for training and 10,000 for testing. It is designed to provide a more challenging task than handwritten digit recognition while maintaining the same format and size as the original MNIST.

[0143] ContinuousFashion: This dataset is obtained by performing a specific transformation on the original FashionMNIST dataset. The image pixel values ​​are converted to continuous values ​​between [-1, 1] instead of the traditional binary values ​​of 0 and 1.

[0144] 3. RL Experiment

[0145] RL experiments are conducted on six benchmarks, including HalfCheetah-v2, BipedalWalker-v3, BipedalWalkerHardcore-v3, Walker2d-v3, LunarLander-v2, and HumanoidStandup-v2. In order to evaluate the long-term performance of the algorithm, the maximum number of time steps is set to 3×10 7The classic proximal policy optimization (PPO) algorithm was chosen as the policy algorithm for gradient optimization. Both the policy network (actor) and the value network (critic) were designed as three-layer perceptron architectures with the Tanh activation function. Due to the different action spaces, the output layer of the policy network used a softmax. The hyperparameter α was tested in both dynamic and constant (initialized to 1) configurations.

[0146] 4. Gradient variance analysis based on CVor

[0147] 4.1 VAE Experiment

[0148] We first compare the variance reduction effect of CVor-based CVs with the baseline and two state-of-the-art (SOTA) methods (Double-CV and RODEO) in a VAE setting. Figure 2 The variance of the gradient estimator using three different control variables (CV) of CVor in VAE experiments compared with the state-of-the-art schemes in four datasets. Figure 2 As shown in Figure 2, the variance reduction effect was tested on four real-world datasets. On the MNIST variant dataset, all CVs-based methods outperformed the baseline methods in terms of variance minimization. Notably, CVs-based CVor performed comparably to the two state-of-the-art methods. On the Continuous Fashion MNIST dataset, the NN-based method achieved the lowest variance, outperforming the state-of-the-art methods. On the Dynamic MNIST, Static MNIST, and Fashion MNIST datasets, the CVor-based AO method achieved lower variance than Double-CV.

[0149] 4.2 RL Experiment

[0150] Since Double-CV and RODEO cannot be reproduced in the RL environment, only the three designs are compared with the baselines on six benchmarks.

[0151] specific Figure 3 The variance of the gradient estimator using three different control variables (CV) of CVor is compared with the baseline in RL experiments on six environments.

[0152] like Figure 3As shown in the figure, the solid line represents the cumulative mean of the variance, while the shaded area represents the instantaneous variance value at each time step. Most CVor-based schemes show lower variance than the baseline. Specifically, in Walker2d-v3, BipedalWalker-v3, and BipedalWalkerHardcore-v3, the RLOO method exhibits the smallest variance; while in the HalfCheetah-v2 environment, the AO method performs better.

[0153] 5. Convergence of gradient optimization based on CVor

[0154] 5.1 VAE Experiment

[0155] specific Figure 4 ELBO evaluation results of three different control variables (CV) of VAE experiments using CVor compared with the state-of-the-art solutions in four datasets. Figure 4 Figure 2 shows the ELBO over time for three CVor designs, two SOTA methods (Double-CV and RODEO), and the baseline method across four datasets. On the MNIST variant dataset, AO and RLOO perform comparably to Double-CV and RODEO; while the NN method performs worse than the four aforementioned methods, it still significantly outperforms the baseline.

[0156] 5.2 RL Experiment

[0157] specific Figure 5 The outcomes of three different forms of rewards in RL experiments are evaluated in six environments compared to a baseline.

[0158] In RL experiments, Figure 5 As shown, the AO method provides the best reward performance in all six benchmarks. In particular, the AO method with adaptive α outperforms the configuration with fixed α = 1. In most cases, the CVor-based method outperforms the baseline. However, the NN method exhibits significant instability: it performs worse than the baseline in BipedalWalker-v3 and performs best in HumanoidStandup-v2. This suggests that the performance of the NN method is highly dependent on the design of the neural network.

[0159] This suggests that neural networks offer a more flexible design paradigm due to their model-free nature and are a direction worth exploring. For example, knowledge distillation or model quantization techniques can be used to obtain the same function as the original function f. η (x) Models with high correlation, thus providing high performance CVs.

[0160] 6. CVor resource assessment

[0161] 6.1. Memory consumption:

[0162] specific Figure 6 Comparison of memory usage under different batch sizes. Figure 6 As shown in Figure 3, the memory usage of three CVor designs, two SOTA methods, and a baseline method is compared. The memory requirements of each method are evaluated for batch sizes of 128, 256, 512, 1024, and 2048.

[0163] Results show that memory usage for AO, NN, RLOO, and the baseline methods is relatively consistent: 155Mb for a batch size of 128, rising to 379Mb as the batch size increases to 2048. In contrast, Double-CV's memory requirement doubles, starting at 248Mb and rising to 822Mb; while RODEO's memory requirement is significantly higher, at 411Mb at the minimum batch size and reaching 4475Mb at a batch size of 2048. Using a CVor-based NN method as CVs, similar to RODEO, does not introduce additional memory overhead.

[0164] 5.6.2 Training time:

[0165] specific Figure 7 Comparison of time complexity under different batch sizes. Figure 7 As shown in Figure 2, the training time of each method is recorded for different batch sizes. The evaluation methods include AO, NN, RLOO, baseline, Double-CV, and RODEO.

[0166] The results show that the training time of AO, NN, RLOO, and the baseline method is similar, and they all show high efficiency at all batch sizes: about 47 minutes for small batches and just over 4 hours for the largest batch size. This proves that the CVor-based solution introduces only minimal additional time overhead.

[0167] Notably, the RODEO method takes significantly longer: 1 hour and 14 minutes for small batches, and a whopping 20 hours and 28 minutes for large batches.

[0168] The computational cost of CVor mainly comes from the complexity of building the gradient calculation graph in the automatic differentiation framework. Table 1 shows the average training time of six environments in 10,000 time steps. The baseline method is used as a reference point. The AO method (fixed f η The performance of the AO method with adaptive α is almost the same as that of the baseline; however, the training time of the AO method with adaptive α is slightly longer, which may be due to the additional computing task of adjusting α.

[0169] Table 1 Average training time (seconds) for six scenarios

[0170]

[0171] The NN-based approach takes longer to train due to the higher computational requirements of forward and backpropagation. For example, in the LunarLander environment, the baseline average training time is 16.07 seconds; after introducing AO (α = 1), it is 17.03 seconds, AO with adaptive α is 17.25 seconds, and NN (adaptive α) reaches 17.50 seconds.

[0172] A similar trend is observed in the Humanoid Standup environment: the baseline is 28.01 seconds, the AO (α = 1) is 28.28 seconds, the AO with adaptive α = 1 is 28.51 seconds, and the NN with adaptive α = 1 is 28.86 seconds. Overall, the training time of the NN method increases significantly, while the AO method with α = 1 is very close to the baseline.

[0173] 7. Evaluation of hyperparameter α: adaptive, static α=1

[0174] We tested the effects of α under both adaptive and static configurations. In all six selected environments, the static configuration with α = 1 performed worse than the adaptive configuration in terms of reward performance, indicating that adaptive hyperparameters can make model training more robust.

[0175] In the CVor framework, α acts as a weight factor to balance between the original gradient and CVs. It is worth noting that when α is set to 0, the CVor solution degenerates into a policy-based optimization.

[0176] specific Figure 8 In addition, the changes of α were observed in six environments, such as Figure 8 The data shows that α decreases rapidly after 2 million iterations and then recovers to a value close to 1 within approximately 5 million steps. For example, in the Humanoid Standup environment, α initially decreases and then increases, with a trend similar to other scenarios, while in the Lunar Lander environment, α remains high, close to 1.

[0177] A gradient estimation device for large model training, used to implement a gradient estimation method for large model training, such as Figure 9 As shown, it includes a control unit, an instruction cache module, a direct memory access module, a data processing module and a data cache module.

[0178] The control unit is used to call instructions from the instruction cache module to control the data processing module;

[0179] The instruction cache module is used to cache sampling instructions, differential instructions and parameter update instructions in response to instruction calls from the control unit;

[0180] The data processing module includes a sampling calculation unit, a gradient estimation unit and a parameter updating unit. The sampling calculation unit is used to perform sampling according to the sampling instruction and calculate the objective function and the parameter update unit according to the sampling sample. The gradient estimation unit is used to perform Cvor gradient estimation and parameter optimization calculation according to the differential instruction, and the parameter updating unit is used to update the model parameters to be optimized according to the parameter update instruction and the Cvor gradient calculation;

[0181] The data cache module includes a data cache unit, a gradient cache unit and a parameter cache unit. The data cache unit is used to cache sampling samples and expected estimation results of the sampling samples, the gradient cache unit is used to cache gradient calculation results, and the parameter cache unit is used to cache updated parameters.

[0182] Specifically, the operation process of a gradient estimation device for large model training implementing a gradient estimation method for large model training is as follows:

[0183] Step 1: Translate the sample {x i}From the distribution p θ (x) Bulk loading

[0184] Control unit: issues sampling instructions.

[0185] Sampling calculation unit: calls the pseudo-random sampler and performs sampling according to the parameter θ.

[0186] Data cache unit: If the model needs to access the external environment or complex data structure sampling, it may trigger DMA (direct memory access module) to load samples from the memory and then temporarily store the samples in the data cache.

[0187] Step 2: Calculation

[0188] Instruction cache module: provides Implementation code.

[0189] Sampling calculation unit: for each x i Call f η (x i ), and then make an expected estimate (such as the mean).

[0190] Data cache module: reads η and sample value cache, and caches function output results.

[0191] Step 3: CVor Loss gradient calculation

[0192] Control unit: calls the CVor expression.

[0193] Gradient Estimation Unit: Performs multiplication Then call the automatic differentiation system (such as PyTorchAutograd) to build a computational graph on the back end and automatically perform differentiation.

[0194] Gradient cache unit: saves CVor gradient results and waits for updates.

[0195] Step 4: Parameter update (θ, η, α)

[0196] Control unit: Generates update instructions according to the preset learning rate rules.

[0197] Parameter update unit: multiplication and subtraction using gradients (SGD / Adam and other operations).

[0198] Parameter cache unit: saves updated parameters; triggers DMA to write updated parameters back to main memory when necessary.

[0199] In another embodiment, a gradient estimation device optimization strategy network training CartPole for large model training

[0200] Goal: Maximize Where x is the trajectory (state-action sequence) generated by the interaction between the agent and the environment; f η (x) is the cumulative return of the estimated trajectory; p θ (x) is the policy network π θ Controlled behavioral distribution; CVor provides a structured gradient optimization method to control variance

[0201] Phase 1: Sampling trajectory x i ~p θ (x)

[0202] Task: The agent interacts with the environment and obtains the trajectory {s0,a0,r0,...,s T}.

[0203] Control unit: issues sampling instructions (executes π θ (a|s) and simulate the environment).

[0204] Sampling calculation unit: performs forward operation and calculates the action probability distribution of each state.

[0205] If using an offline dataset, triggers a DMA bulk load state sequence.

[0206] Data cache unit: temporarily stores states, actions, and reward sequences for subsequent processing.

[0207] Phase 2: Calculate f η(x) - Estimated returns (such as GAE)

[0208] Instruction cache module: load critic evaluation code or function f η (x).

[0209] Sampling calculation unit: runs the critic network and estimates the value function of the trajectory calculation.

[0210] Data cache unit: record f η (x i )result.

[0211] Phase 3: Constructing CVor benchmark items and CVor Loss

[0212] Gradient Estimation Unit: Construction Post-calculation

[0213] Gradient cache unit: storage Wait for the results.

[0214] Stage 4: Backpropagation CVor Loss to get gradient

[0215] Control unit: calls the automatic differentiation system.

[0216] Gradient estimation unit: constructs a computational graph and propagates errors step by step.

[0217] Data caching module: caches gradient paths and weights of each layer to speed up calculations.

[0218] Phase 5: Update parameters

[0219] Parameter update unit: calculates and completes parameter updates.

[0220] Data cache module: maintains updated new parameters.

[0221] DMA can be triggered to write the updated model weights back to global storage or video memory.

[0222] The above-described embodiments merely represent specific implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of protection of the present application. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the technical concept of the present application, and all such variations and improvements fall within the scope of protection of the present application.

Claims

1. A gradient estimation operator for large model training, characterized in that: Apply the control variable to the original function, indirectly change the result of the random calculation graph, and change the loss function The gradient map from Convert to It is expressed as follows: f η (x) represents the proxy loss function, h(x) represents the control variable function corresponding to each sample x, a is a real number in (0,1], is a dynamically adjusted baseline or control variable function, for The expected form of θ depends on θ and has the same gradient expectation with respect to θ. exp(·) is the exponential function, ⊙ refers to the Magic-Box operator, represents the expected return, x k is the input sample, α is a scalar hyperparameter, p θ (x) is the probability distribution, is the gradient calculation, is the loss function, is the gradient estimation operator, and K is the number of samples.

2. A gradient estimation operator for large model training according to claim 1, characterized in that: In a near-optimal design: Where, It is a low-order linear approximation of f(x), obtained by linear regression fitting.

3. A gradient estimation operator for large model training according to claim 1, characterized in that: In RLOO:

4. A gradient estimation operator for large model training according to claim 1, characterized in that: In DoubleCV: Where b(x) is f η First-order Taylor approximation of (x).

5. A gradient estimation operator for large model training according to claim 1, characterized in that: In a neural network model parameterized by parameter φ: Where h φ (x,f η (x)) represents a function from a family of flexible parameters with parameter φ.

6. A gradient estimation method for large model training, characterized in that: Utilize the gradient estimation operator optimization for large model training according to any one of claims 1-5 include: Step 1: Input the target function f η (x), distribution p θ (x), aggregate function and Initialize hyperparameters θ, η, α = 0; Step 2: For i=1,2,…, from p θ (x) extract K samples {x i } i∈{1,…,K} , that is, x i ~p θ (x); Step 3: Calculation Step 4: Find the gradient of CVor loss: Step 5: Calculate CVor gradient Step 6: Update the hyperparameters θ, η, and α.

7. A gradient estimation method for large model training according to claim 6, characterized in that: Update the parameters θ which is equivalent to the objective function:

8. A gradient estimation method for large model training according to claim 6, characterized in that: Update the learning rate η:

9. A gradient estimation method for large model training according to claim 6, characterized in that: Update the hyperparameter α:

10. A gradient estimation device for large model training, characterized in that: A method for implementing a gradient estimation method for large model training according to claim 6, comprising a control unit, an instruction cache module, a direct memory access module, a data processing module, and a data cache module, The control unit is used to call instructions from the instruction cache module to control the data processing module; The instruction cache module is used to cache sampling instructions, differential instructions and parameter update instructions in response to instruction calls from the control unit; The data processing module includes a sampling calculation unit, a gradient estimation unit and a parameter updating unit. The sampling calculation unit is used to perform sampling according to the sampling instruction and calculate the objective function and the parameter update unit according to the sampling sample. The gradient estimation unit is used to perform Cvor gradient estimation and parameter optimization calculation according to the differential instruction, and the parameter updating unit is used to update the model parameters to be optimized according to the parameter update instruction and the Cvor gradient calculation; The data cache module includes a data cache unit, a gradient cache unit and a parameter cache unit. The data cache unit is used to cache sampling samples and expected estimation results of the sampling samples, the gradient cache unit is used to cache gradient calculation results, and the parameter cache unit is used to cache updated parameters.

Images