An output layer perturbation variance reduction likelihood ratio neural network training method and system

Abstract

This invention discloses a method and system for training a likelihood ratio neural network with reduced output layer perturbation variance. The method includes: acquiring training data, neural network parameters, and output dimension; setting the noise scale, number of samples, and batch size; performing a deterministic forward propagation calculation using the neural network; copying the initial output vector multiple times and introducing noise to form perturbed output samples, and obtaining their corresponding loss values; constructing the gradient signal of the output layer using the likelihood ratio formula; and propagating the gradient signal of the output layer back to the network parameters through the vector-Jacobi product mechanism of the automatic differentiation framework, calculating the parameter gradients, and updating the neural network parameters. By implementing this invention, an effective training direction is provided under real black-box functions and without relying on the differentiability of the loss function, while avoiding high-dimensional space perturbation sampling, reducing variance, improving computational efficiency, seamlessly integrating into the automatic differentiation framework, and ensuring training stability and scalability in various complex scenarios.

Description

Technical Field

[0001] This invention relates to neural networks, and more specifically to a method and system for training neural networks with reduced likelihood ratios due to output layer perturbation variance. Background Technology

[0002] Backpropagation is the cornerstone of modern neural network training. It calculates the gradient of the loss function with respect to the network parameters using the chain rule. However, this method relies on the differentiability of the loss function with respect to the network output and requires a stable and valid gradient signal. Traditional backpropagation methods encounter challenges when dealing with non-differentiable or black-box objectives such as hard 0-1 loss, piecewise constant loss, and truncated loss. These objectives may not provide continuous gradient information, making the optimization process difficult.

[0003] To overcome these limitations, likelihood ratio methods and REINFORCE-like methods have been proposed, which can estimate gradients without relying on the differentiability of the objective function. However, these methods typically introduce random perturbations into the parameter or input space, which can lead to increased estimation variance and convergence instability in high-dimensional cases. Similarly, evolutionary strategies and zero-order optimization, while capable of handling black-box objectives, are limited by their high computational cost. Furthermore, although forward learning or likelihood ratio methods have attempted to mitigate these problems, their inherent stochastic mechanisms often amplify variance during propagation in deep networks, impacting training efficiency.

[0004] Therefore, it is necessary to design a new method to provide effective training directions in real black-box functions and without relying on the differentiability of loss functions, while avoiding high-dimensional space perturbation sampling, reducing variance and improving computational efficiency, seamlessly integrating into automatic differentiation frameworks, and ensuring training stability and scalability in various complex scenarios. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for training neural networks with reduced output layer perturbation variance and likelihood ratio.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for training a neural network with reduced output layer perturbation variance, comprising:

[0007] Obtain training data, neural network parameters, and output dimensions; set noise scale, number of samples, and batch size to obtain input data. The input data is subjected to a deterministic forward propagation computation using a neural network to obtain an initial output vector; The initial output vector is copied multiple times, and noise is introduced to form perturbed output samples. The corresponding loss value is obtained for each perturbed output sample. Based on the loss value and the perturbed output sample, the gradient signal of the output layer is constructed using the likelihood ratio formula; The gradient signal of the output layer is backpropagated to the network parameters through the vector-Jacobi product mechanism of the automatic differentiation framework, and the parameter gradient is calculated. The neural network parameters are then updated based on the parameter gradient.

[0008] The further technical solution is as follows: The initial output vector is copied multiple times, and noise is introduced to form perturbed output samples. For each perturbed output sample, its corresponding loss value is obtained, including: The initial output vector is copied multiple times, noise is sampled according to the set noise scale, and the inverse noise is generated for half of the multiple vectors to obtain the perturbed output sample; Obtain the corresponding loss value for each perturbation output sample.

[0009] The further technical solution is as follows: obtaining the corresponding loss value for each perturbation output sample includes: For each perturbation output sample, the loss value is calculated by calling the target loss function or the black-box evaluation function.

[0010] The further technical solution is as follows: the neural network includes CNN, ResNet, DenseNet, ViT, Transformer, BERT, Mamba, GAN, VAE, ACT or other neural networks.

[0011] The further technical solution is that the perturbation output sample includes Gaussian noise or other symmetrical noise.

[0012] The further technical solution is as follows: The initial output vector is copied multiple times, noise is sampled according to a set noise scale, and the inverse noise is generated for half of the multiple vectors to obtain perturbed output samples, including: For each vector, a set of noise values ​​is first randomly sampled according to the noise scale; According to the dual sampling principle of the output stage, if the set number of samples is even, then for each pair of positive and negative perturbations, the original output is constructed by adding noise and the original output is subtracted from the same noise to obtain the perturbation output sample.

[0013] The present invention also provides a neural network training system for reducing the likelihood ratio of the output layer perturbation variance, comprising: The acquisition unit is used to acquire training data, neural network parameters, noise scale, number of samples, batch size, and output dimension to obtain input data; A forward propagation computation unit is used to perform a deterministic forward propagation computation on the input data using a neural network to obtain an initial output vector; The sample processing unit is used to copy the initial output vector multiple times, introduce noise to form perturbed output samples, and obtain the corresponding loss value for each perturbed output sample. A signal construction unit is used to construct the gradient signal of the output layer based on the loss value and the perturbed output sample using the likelihood ratio formula. The update unit is used to backpropagate the gradient signal of the output layer back to the network parameters through the vector-Jacobi product mechanism of the automatic differentiation framework, calculate the parameter gradient, and update the neural network parameters according to the parameter gradient.

[0014] The further technical solution is as follows: the sample processing unit includes: The perturbation processing subunit is used to copy the initial output vector multiple times, sample noise according to the set noise scale, and generate the inverse noise of half of the multiple vectors to obtain the perturbation output sample. The loss value calculation subunit is used to obtain the corresponding loss value for each perturbation output sample.

[0015] The further technical solution is as follows: the loss value calculation subunit is used to calculate the loss value of each perturbation output sample by calling the target loss function or the black box evaluation function.

[0016] The further technical solution is as follows: the disturbance processing subunit includes: The random sampling module is used to randomly sample a set of noise values ​​for each vector according to the noise scale. The noise addition / reduction module is used to construct the original output plus noise and the original output minus the same noise respectively for each pair of positive and negative perturbations, according to the dual sampling principle of the output stage. If the set number of samples is even, the perturbation output sample is obtained.

[0017] The advantages of this invention compared to existing technologies are as follows: After obtaining the initial output vector through a single forward propagation, this invention copies it multiple times and introduces noise to form perturbation samples. Based on these samples and their loss values, a gradient signal is constructed using the likelihood ratio formula. Then, the gradient signal is backpropagated via the vector-Jacobi product mechanism of the automatic differentiation framework to update the network parameters. This method cleverly provides an effective training direction without relying on the differentiability of the loss function, while avoiding the high variance problem caused by directly performing perturbation sampling in high-dimensional space. By performing operations at the output layer, the computational requirements are significantly reduced, improving efficiency. Furthermore, it is seamlessly integrated into existing automatic differentiation frameworks, ensuring training stability and scalability in complex scenarios such as hard 0-1 loss, labeled noise, and black-box optimization.

[0018] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a neural network training method for reducing the likelihood ratio of the output layer perturbation variance, provided in an embodiment of the present invention. Figure 2 A schematic diagram illustrating the differences between standard backpropagation, the traditional likelihood ratio method, and the method of the present invention, provided for embodiments of the present invention; Figure 3 This is a schematic diagram illustrating the variation of the variance of the gradient estimation with noise scale in the combination of output-level repeated sampling and dual sampling, provided in an embodiment of the present invention. Figure 4 A schematic diagram illustrating the convergence speed and final accuracy provided in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the generation of effective gradient estimation from output perturbation, provided in an embodiment of the present invention. Figure 6 A schematic diagram illustrating the change in stability provided in an embodiment of the present invention; Figure 7 A schematic diagram of the error region in a robust regression task provided by an embodiment of the present invention; Figure 8 This is a schematic block diagram of a neural network training system for reducing the likelihood ratio of the output layer perturbation variance, provided in an embodiment of the present invention. Figure 9 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0023] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0024] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for training a neural network with reduced output layer perturbation variance according to an embodiment of the present invention. This method is applied in a server. After acquiring training data and related parameters, forward propagation is performed to obtain an initial output vector. An innovative "output-level dual sampling" strategy is employed to copy and introduce positive and negative noise to form perturbation samples, offsetting some random noise terms and reducing variance in the estimation without relying on the differentiability of the loss function. Then, based on these perturbation samples and the corresponding loss values, gradient signals are constructed using the likelihood ratio formula. The parameter gradients are then efficiently calculated through backpropagation using an automatic differentiation framework, achieving model updates. This method avoids complex perturbation sampling directly in high-dimensional space, reducing computational complexity and improving computational efficiency. Simultaneously, it seamlessly integrates into existing automatic differentiation frameworks, ensuring the stability and scalability of the training process in complex scenarios, including but not limited to CNNs and ResNets, effectively improving the effectiveness and reliability of neural network training.

[0026] Figure 1 This is a schematic flowchart of the neural network training method for reducing the likelihood ratio by reducing the output layer perturbation variance provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110 to S150.

[0027] S110. Obtain training data, neural network parameters, and output dimension; set noise scale, number of samples, and batch size to obtain input data.

[0028] In this embodiment, input data refers to datasets collected or generated (e.g., images, text, sensor data, etc.), combined with predefined neural network parameters (including weights and biases), a specified noise scale (controlling the perturbation intensity), a sampling number (the number of times the initial output vector is copied and noise is introduced after each forward propagation), a batch size (the number of data samples processed at one time), and an output dimension (the dimension of the final output layer of the neural network). These elements together form the basis for subsequent computations. In particular, this step ensures that all necessary information is correctly prepared to facilitate an effective training process.

[0029] S120. Perform a deterministic forward propagation calculation on the input data using a neural network to obtain an initial output vector.

[0030] In this embodiment, the initial output vector refers to the raw logits generated by the output layer of the neural network after the input data undergoes a complete forward propagation calculation through the neural network model. Specifically, the input data first undergoes a series of linear transformations (such as matrix multiplication) and nonlinear activation functions, passing layer by layer until it reaches the output layer, producing a task-related predicted value or feature representation. For classification tasks, this might be a vector representing the probability distribution of each class; while for regression tasks, it might be the directly predicted target variable value. Importantly, the emphasis here is on "determinism," meaning that for a given input... and current network parameters Output vector The fact that the output vector is a uniquely determined real-valued vector (a non-random variable) ensures that the output vector is uniquely determined based on the current parameters. This initial output vector is then used as the basis for constructing perturbation samples, further supporting the likelihood ratio gradient estimation process.

[0031] The neural network includes CNN, ResNet, DenseNet, ViT, Transformer, BERT, Mamba, GAN, VAE, ACT, or other neural networks.

[0032] S130. Copy the initial output vector multiple times and introduce noise to form perturbed output samples, and obtain the corresponding loss value for each perturbed output sample.

[0033] In this embodiment, perturbation output samples refer to a dataset generated by repeatedly copying the initial output vector and adding different types of random noise (such as Gaussian noise or other forms of symmetric noise) to each copy. This process aims to estimate the gradient signal by introducing controllable random perturbations, thereby providing effective guidance for neural network training. The perturbation output samples not only contain information predicted by the original model but also incorporate changes introduced by noise, allowing subsequent steps to construct the gradient signal of the output layer by calculating the difference between these perturbation outputs and the target (i.e., the loss value).

[0034] The perturbation output samples include Gaussian noise or other symmetrical noise.

[0035] In one embodiment, step S130 described above may include steps S131 to S132.

[0036] S131. Copy the initial output vector multiple times, sample noise according to the set noise scale, and generate the opposite noise for half of the multiple vectors to obtain the perturbed output sample.

[0037] In this embodiment, to construct perturbation output samples, the initial output vector first needs to be copied. Specifically, each copied vector represents a possible output state, and by superimposing different noises on these copies, multiple possible scenarios can be simulated. Next, according to a pre-set noise scale, a set of noise values ​​is randomly sampled for each copied vector. This is done to ensure that the perturbation has a certain degree of control and consistency, avoiding excessively large or small noises from affecting the training effect. In addition, when the set number of samples is even, the opposite noise is generated specifically for half of the vectors. This is based on the principle of "output-level dual sampling," aiming to cancel out some random noise terms through positive and negative perturbations, further reducing the variance of the likelihood ratio gradient estimation.

[0038] In one embodiment, step S131 may include steps S1311 to S1312.

[0039] S1311. For each vector, a set of noise values ​​is randomly sampled according to the noise scale.

[0040] In this embodiment, in this refinement step, for each replicated initial output vector, a set of noise values ​​is independently extracted from a specific distribution (e.g., a standard normal distribution) according to a preset noise scale parameter. This step is crucial for implementing random perturbation of the output layer, directly determining the degree and nature of the perturbation. The choice of noise scale is critical because it directly affects the perturbation intensity and the quality of the resulting gradient signal. Appropriately adjusting the noise scale can help balance exploration and exploitation, ensuring that the training process can effectively explore the solution space without causing performance degradation due to excessive perturbation.

[0041] S1312. According to the dual sampling principle of the output stage, if the set number of samples is even, then for each pair of positive and negative perturbations, the original output is constructed by adding noise and the original output is subtracted from the same noise to obtain the perturbation output sample.

[0042] In this embodiment, when the number of samples is set to an even number, a dual sampling strategy at the output level is adopted. This means that when generating perturbed output samples, instead of simply adding random noise to each replicated initial output vector, it is processed in pairs: for each pair of perturbed samples, one pair adds a certain noise to the original output, and the other subtracts the same noise. This approach helps reduce the variance caused by randomness, because positive and negative perturbations can cancel each other out to some extent, thus making the final gradient estimate more stable and reliable.

[0043] S132. Obtain the corresponding loss value for each perturbation output sample.

[0044] Specifically, for each perturbation output sample, the loss value is calculated by calling the target loss function or the black-box evaluation function.

[0045] In this embodiment, after constructing the perturbation output samples, the next step is to call the target loss function or black-box evaluation function for each perturbation output sample to calculate its respective loss value. The key here is that even if the loss function itself is non-differentiable or a completely unknown black-box function, as long as it returns a specific numerical result, it can be used to construct the training signal. In this way, the method of this embodiment can still provide an effective parameter update direction for the neural network without relying on the traditional backpropagation mechanism. This step is not only one of the core links in the entire process, but also the foundation for effectively training complex tasks (such as robust classification with labeled noise, sparse success / failure reward optimization, etc.).

[0046] Calculates the loss or score value for the perturbation output. This module can be a differentiable loss, a hard 0-1 loss, a truncated loss, an external simulator, a rule system, or a black-box function that only returns a scalar.

[0047] S140. Based on the loss value and the perturbed output sample, construct the gradient signal of the output layer using the likelihood ratio formula.

[0048] In this embodiment, after calculating the loss value for each perturbed output sample, the next step is to construct the gradient signal of the output layer using these loss values ​​and the corresponding perturbed output. This process relies on the likelihood ratio method, which allows for effective estimation of the gradient signal even when the loss function is non-differentiable or cannot be integrated into the computational graph. Specifically, for each perturbed output sample and its corresponding loss value, a gradient signal can be estimated using the likelihood ratio formula. This gradient signal essentially reflects the direction and intensity of the output layer's change relative to the input noise, thus providing a basis for subsequent parameter updates. The application of the likelihood ratio formula can not only handle non-differentiable loss functions but also effectively reduce variance in high-dimensional parameter spaces, making gradient estimation more stable and reliable. Furthermore, by combining output-level resampling and dual sampling strategies, the uncertainty caused by random noise is further reduced, ensuring the accuracy and consistency of the gradient signal.

[0049] S150. Using the vector-Jacobi product mechanism of the automatic differentiation framework, the gradient signal of the output layer is backpropagated to the network parameters, and the parameter gradient is calculated. The neural network parameters are then updated based on the parameter gradient.

[0050] In this embodiment, once the gradient signal of the output layer is obtained, the next task is to backpropagate it back to the network parameters for parameter updates. This process leverages the automatic differentiation mechanism in modern deep learning frameworks, particularly the Vector-Jacobian Product (VJP). The VJP mechanism allows us to directly calculate the gradient with respect to the network parameters from the gradient signal of the output layer without recalculating the entire forward propagation path. This significantly improves efficiency, especially when dealing with complex neural network structures. Specifically, this process can be implemented by calling PyTorch's `Z.backward(gradient=v)`, JAX's `vjp` interface, or the corresponding functions in TensorFlow. The obtained parameter gradients are then passed to an optimizer (such as SGD, Adam, etc.) to update the weights of the neural network. This series of operations not only ensures the efficiency of the training process but also demonstrates good adaptability when facing non-differentiable loss functions or external evaluation metrics. Finally, after multiple iterations, the parameters of the neural network are gradually adjusted to minimize the loss function and improve model performance. This step is one of the core components of the entire training process, directly determining the model's learning and generalization abilities. In this way, this embodiment provides a novel and efficient training method that solves the challenges faced by traditional methods when dealing with non-differentiable loss functions or other complex scenarios.

[0051] For example, the input is a mini-batch of data (X, Y), where X is the dataset, Y is the label set corresponding to the dataset, and the neural network parameters... Noise scale The sample size n, the batch size B, and the output dimension d.

[0052] Perform a deterministic forward propagation: Z is the Logits output of the model (an unnormalized real vector output by the last fully connected layer of the neural network). ; Will Copy Y to n copies, and simultaneously copy Y to n copies. ,get One output evaluation sample.

[0053] Half of the noise before sampling , Indicates noise distribution. Represents a d-dimensional normal distribution, with the second half of the noise set to its negative: ; Construct perturbation output: ; Batch black-box loss evaluation of the perturbation output: ; Constructing vectorized output layer gradient signals: ; in, This represents the element-wise or sample-by-sample broadcast product of the loss value and the corresponding noise sample. In practice, it can be repeated along the original batch dimension. Summing or averaging the samples makes Consistent with the output Z-shape.

[0054] Calling the vector-Jacobi product: ; This step is equivalent to calling the external gradient backpropagation on Z in the automatic differentiation framework.

[0055] Optimizer based on Update parameters and return the updated value. .

[0056] In this embodiment, unlike perturbations in parameters, inputs, or hidden layers, noise is introduced only in the low-dimensional output layer: ; in, For output layer noise samples, For noise scale. The system outputs for each disturbance. The loss value is obtained by calling the target loss function or the black-box evaluation function. Even if the loss function is non-differentiable or cannot be integrated into the computation graph, as long as it can return a numerical value, this embodiment can still use its numerical output to construct a training signal.

[0057] Based on the likelihood ratio formula, the gradient signal of the output layer can be estimated as follows: Where n is the number of repeated samples in the output layer. Subsequently, the system uses v as the external gradient of the output Z and calculates the parameter gradients using the vector-Jacobi product of the automatic differentiation framework: ; This step can be implemented using PyTorch's Z.backward(gradient=v), JAX's vjp, TensorFlow's external gradient mechanism, or equivalent interfaces. Therefore, the loss function itself does not need to be differentiable, nor does it need to store the computational graph of the loss function.

[0058] To reduce the variance of the likelihood ratio gradient estimation, this embodiment introduces two core variance reduction mechanisms: Output-level resampling: The same deterministic output Z is copied multiple times, and different noises are superimposed only at the output layer to repeatedly evaluate the loss, thus avoiding repeated forward propagation of the network backbone.

[0059] Output stage dual sampling: When n is even, sample first. , and then This cancels out some of the random noise terms through positive and negative perturbations.

[0060] In this embodiment, the variance reduction process involves output-level resampling, dual sampling, optional baseline subtraction, loss normalization, or noise scaling. The parameter optimization process updates the network parameters using SGD, Adam, AdamW, RMSProp, or other optimizers.

[0061] In this embodiment, the loss function does not need to be differentiable with respect to the output, nor does it need to be connected to the computation graph. The output layer gradient signal is fed into the automatic differentiation framework as an external gradient to calculate the Jacobian product of the output with respect to the parameters. Hard 0-1 loss or mixed loss is used, and this loss is optimized through output layer likelihood ratio estimation. A truncated loss is used, and the zero gradient problem in the truncated region is overcome by perturbation of the output layer.

[0062] The key efficiency advantage of this embodiment lies in decoupling the network backbone from the stochastic estimation process. Standard backpropagation requires one model forward pass, one loss forward pass, one model backward pass, and one loss backward pass. Traditional input-level resampling or hidden-layer perturbation methods typically require n model forward passes and a high intermediate activation buffer. In contrast, this embodiment only performs one deterministic model forward pass and one VJP pass. As the number of samples n increases, it mainly adds low-cost output-layer perturbation, numerical evaluation of loss, and gradient signal construction, as shown in Table 1.

[0063] Table 1. Comparison Results

[0064] in, This represents the forward cost of the model backbone. Indicates the cost of a single loss assessment. This indicates that the model is being activated and cached. Since this embodiment treats the loss function as a black-box numerical function, there is no need to cache the loss function computation graph, making it particularly suitable for non-differentiable or externally evaluated functions.

[0065] Please see Figure 2 Standard backpropagation relies on a complete differentiable computational graph; traditional likelihood ratio methods typically require multiple complete forward propagations; this embodiment restricts the perturbation to the output layer and preserves the efficiency of a single forward propagation through VJP.

[0066] Please see Figure 3 This indicates that under the combination of output-level repeated sampling and dual sampling, the gradient estimation variance is more stable to noise scale changes, and the dominant noise is significantly suppressed.

[0067] Please see Figure 4 This demonstrates that in a standard supervised task with a differential signal, the convergence speed and final accuracy of this embodiment can approach those of standard backpropagation.

[0068] Please see Figure 5 This indicates that when there is a zero gradient region in the truncated loss, standard backpropagation may not be able to recover the true signal, while this embodiment generates an effective gradient estimate by output perturbation.

[0069] Please see Figure 6 This indicates that under high label noise, directly optimizing the 0-1 target, the method in this embodiment is more stable than the cross-entropy baseline.

[0070] Please see Figure 7 This indicates that within a reasonable noise scale and truncation threshold range, this embodiment has a wide low-error region in robust regression tasks.

[0071] In variance reduction ablation experiments, ResNet-18 was used on CIFAR-10, CIFAR-100, and TinyImageNet to compare the effects of ordinary perturbations, dual sampling, resampling, and noise scaling. Experiments show that both dual sampling and output-level resampling improve stability; when combined and the number of resampling operations is high... At that time, the model's sensitivity to noise scale Significantly reduced. As shown in Table 2.

[0072] Table 2. Experimental Results

[0073] On CIFAR-100 and TinyImageNet, ResNet-18, DenseNet-121, and ViT-B / 16 were used to compare the standard backpropagation (BP) method with the method of this embodiment. The results show that, in standard differentiable tasks, the accuracy of the method of this embodiment and BP is generally within about 1%, with only a small increase in training time per round; as shown in Table 3.

[0074] Table 3. Comparison of accuracy and time consumption between backpropagation and standard method

[0075] In the SimpleCNN task of MNIST, the methods of BP, this embodiment, GLR-style input-level repetition, and parameter-level vanilla LR are compared, as shown in Table 4.

[0076] Table 4. Comparison with LR family methods

[0077] This experiment directly supports the computational and memory advantages of output-level perturbations compared to input-level or parameter-level perturbations.

[0078] Hard 0-1 loss is a piecewise constant objective, and standard backpropagation can hardly provide effective gradients. On CIFAR-10, standard backpropagation cannot train with hard 0-1 loss, while the method of this invention can achieve Top-1 accuracy of 61.74%, Top-3 accuracy of 76.47%, and Top-5 accuracy of 79.39% with hard 0-1 loss. If a hybrid loss is used, the Top-1 accuracy on CIFAR-10 can reach 87.92%, as shown in Table 5.

[0079] Table 5. Hard 0-1 loss comparison

[0080] This experiment supports the method in this embodiment's ability to directly optimize non-differentiable targets with vanishing gradients.

[0081] Under CIFAR-10 symmetric label noise, the method in this embodiment directly optimizes the 0-1 target, which is more stable under high noise compared to the cross-entropy baseline. As shown in Table 6.

[0082] Table 6. Label Noise Robust Classification

[0083] This experiment supports the effectiveness of the design example in avoiding overfitting to noisy labels using bounded 0-1 loss.

[0084] 20% outliers and abnormal offsets In a one-dimensional sine regression task, truncation The loss exhibits a zero gradient problem in outlier regions. Experimental results are shown in Table 7.

[0085] Table 7. Robust Regression Experiment of Cutoff Loss

[0086] This experiment supports the effectiveness of the method in this embodiment in expanding the "gradient capture range" of the truncation loss through Gaussian smoothing and likelihood ratio estimation.

[0087] In experiments on black-box optimization, generative models, language models, and robotic tasks, the method presented in this embodiment demonstrated its unique advantages and high performance across various application scenarios. Firstly, in the discrete pseudo-Boolean optimization (IOH discrete PBO) black-box optimization task, the method achieved a normalized score of 0.9917 and a success rate of 86.7%, significantly outperforming the REINFORCE algorithm's score of 0.7040 and 0% success rate. This indicates that when dealing with complex, non-differentiable optimization problems, the method presented in this embodiment can provide a more accurate and reliable solution.

[0088] For continuous black-box optimization tasks (IOH continuous BBOB), this embodiment also performs excellently, achieving a normalized score of 0.2389 and a target hit rate of 25.0%, which outperforms the Gaussian-REINFORCE algorithm. This demonstrates the superiority of this embodiment in handling high-dimensional, nonlinear problems and the effectiveness of its variance reduction strategy.

[0089] In terms of generative models, the method in this embodiment demonstrated good performance for both GAN and VAE models. Particularly in the MNIST GAN generation task, the method achieved an FID score of 40.27, significantly outperforming the traditional backpropagation (BP) method's 53.33. While the FID scores of the method in this embodiment were close to those of the BP method in the VAE generative model test (29.80 and 29.73 respectively), the method still demonstrated its adaptability and competitiveness in different scenarios.

[0090] Furthermore, the method described in this embodiment also demonstrates its wide applicability in the Mamba selective replication task and the BERT-based IMDB sentiment classification task. In the Mamba task, both the method described in this embodiment and the BP method achieved a validation accuracy of 99.8%, proving the effectiveness of this embodiment in handling complex decision-making tasks. While the accuracy of the method described in this embodiment is 91.91%, slightly lower than the BP method's 92.43%, the training time for both is similar, indicating that the method described in this embodiment maintains high efficiency while ensuring performance.

[0091] Finally, in the Aloha dual-arm robot task, the overall average success rate of the method in this embodiment reached 48.0%, higher than the 47.0% of the BP method. This result further demonstrates the potential and practicality of the method in real-world physical manipulation tasks, especially in situations requiring precise control and stability.

[0092] In summary, the method of this embodiment has demonstrated superior performance in multiple fields and tasks, not only solving the challenges faced by traditional methods, but also providing new directions for future research and applications.

[0093] The theoretical basis of this embodiment is to set the output random variable as: ; For expected loss Even if L is not differentiable, as long as L is bounded or satisfies appropriate conditions, the desired target after Gaussian convolution will still have smoothness. Using the likelihood ratio technique, we can obtain: ; Output-level repeated sampling reduces the estimated variance by the number of samples n; dual sampling utilizes the symmetry of the noise distribution to partially cancel out the dominant noise term. Convergence analysis shows that, under appropriate step size and bounded variance conditions, the method in this embodiment can converge to the stable point of the smoothing objective; when the step size is... During decay, the convergence rate can approach .

[0094] For hard 0-1 loss, the values ​​are restricted to 0 and 1, so the gradient estimation magnitude is controlled by the noise score function and will not produce excessive gradients due to mislabeling, unlike cross-entropy. For truncated loss, Gaussian convolution smooths the originally zero-gradient "dead zone" into a region that still has a non-zero gradient, allowing the model to recover from the initial prediction falling into the truncated region.

[0095] In this embodiment, the number of repeated samples n in the output layer can be 100, 200, 300 or higher in CIFAR-10 / 100 / TinyImageNet; the language model example uses 200. The output noise scale is mostly set to 1.0 in experiments; however, it can also be adjusted within the range of 0.01 to 5.0. Noise distribution refers to the perturbation distribution of the output layer. Zero-mean Gaussian noise distribution is preferred, but symmetrical noise distribution can also be used. Dual sampling refers to whether or not to use and- Preferred selection enabled; Loss evaluation refers to differentiable or non-differentiable objectives, including cross-entropy, hard 0-1, truncation, simulator scoring, etc. Automatic differentiation interface refers to VJP implementation methods, including backward (gradient=v), VJP, or equivalent interfaces; The preheating strategy refers to stabilizing the initial convergence. In the label noise experiment, five rounds of cross-entropy preheating were used before switching to 0-1 optimization. The optimizer refers to the parameter update method, such as Adam, AdamW, SGD, etc.

[0096] The method in this embodiment is applicable not only to various noise distributions, including but not limited to Gaussian, Laplace, Student's t, Rademacher, and uniform zero-mean symmetric distributions, but also to different types of perturbation objects, such as classification logits, regression outputs, action means, and generator outputs. Regarding variance reduction strategies, in addition to dual sampling, baseline subtraction and control variables can be combined to achieve lower gradient estimation variance.

[0097] Furthermore, the method in this embodiment can be used alone to replace traditional backpropagation (BP), or it can be used in combination with standard BP, for example, by weighted fusion of cross-entropy gradients and gradients generated by the method in this embodiment. Regarding the choice of loss function, this embodiment is not limited to locally defined functions, and also supports numerical scores returned by external black-box programs, simulators, or online evaluation services as optimization targets.

[0098] This method offers significant technical advantages: First, it can directly optimize non-differentiable, piecewise constant, vanishing gradient, or black-box targets. Second, by perturbing the output layer, whose dimensionality is typically much lower than the parameter space or hidden layer space, it achieves lower gradient estimation variance. Notably, when increasing the number of samples, the network backbone only needs to perform one deterministic forward propagation and one vector-Jacobi product (VJP) calculation per iteration, greatly improving efficiency. Simultaneously, the output-level resampling and dual sampling mechanisms further reduce variance, making the training process more stable and more robust to noise scale variations.

[0099] This embodiment is particularly suitable for handling challenging scenarios such as non-differentiable loss functions, label noise, outliers, and truncated losses, and demonstrates significant advantages in areas such as hard 0-1 loss classification, black-box optimization, generative model training, language model training, and robot imitation learning. Especially in addressing key issues in neural network training, such as how to provide effective training directions when the loss function is non-differentiable, and how to avoid perturbation sampling in high-dimensional space to reduce the variance of likelihood ratio estimation, it provides practical solutions.

[0100] In summary, this embodiment aims to provide an efficient and stable training method and system for solving complex problems in neural network training by introducing a gradient signal transmission mechanism based on output layer random perturbation and likelihood ratio method, combined with advanced variance reduction technology. It is applicable to a wide range of machine learning tasks and application scenarios.

[0101] The aforementioned output layer perturbation variance reduction likelihood ratio neural network training method obtains the initial output vector through a single forward propagation, then copies it multiple times and introduces noise to form perturbation samples. Based on these samples and their loss values, a gradient signal is constructed using the likelihood ratio formula. This gradient signal is then backpropagated via the vector-Jacobi product mechanism of the automatic differentiation framework to update the network parameters. This method cleverly provides an effective training direction without relying on the differentiability of the loss function, while avoiding the high variance problem caused by directly performing perturbation sampling in high-dimensional space. By performing operations at the output layer, it significantly reduces computational requirements, improves efficiency, and seamlessly integrates into existing automatic differentiation frameworks, ensuring training stability and scalability in complex scenarios such as hard 0-1 loss, labeled noise, and black-box optimization.

[0102] Figure 8 This is a schematic block diagram of a neural network training system 300 for reducing the likelihood ratio of the output layer perturbation variance, provided in an embodiment of the present invention. Figure 8 As shown, corresponding to the above-described method for training neural networks with reduced output layer perturbation variance, this invention also provides a neural network training system 300 for reduced output layer perturbation variance. This neural network training system 300 includes units for executing the above-described method for training neural networks with reduced output layer perturbation variance, and the system can be configured in a server. Specifically, please refer to... Figure 8 The output layer perturbation variance reduction likelihood ratio neural network training system 300 includes an acquisition unit 301, a forward propagation calculation unit 302, a sample processing unit 303, a signal construction unit 304, and an update unit 305.

[0103] The acquisition unit 301 is used to acquire training data, neural network parameters, noise scale, number of samples, batch size, and output dimension to obtain input data; the forward propagation calculation unit 302 is used to perform a deterministic forward propagation calculation on the input data using the neural network to obtain an initial output vector; the sample processing unit 303 is used to copy the initial output vector multiple times, introduce noise to form perturbed output samples, and obtain the corresponding loss value for each perturbed output sample; the signal construction unit 304 is used to construct the gradient signal of the output layer based on the loss value and the perturbed output samples using the likelihood ratio formula; the update unit 305 is used to backpropagate the gradient signal of the output layer back to the network parameters through the vector-Jacobi product mechanism of the automatic differentiation framework, calculate the parameter gradient, and update the neural network parameters according to the parameter gradient.

[0104] In one embodiment, the sample processing unit 303 includes: a perturbation processing subunit, used to copy the initial output vector multiple times, sample noise according to a set noise scale, and generate the inverse noise of half of the multiple vectors to obtain perturbation output samples; and a loss value calculation subunit, used to obtain the corresponding loss value for each perturbation output sample.

[0105] In one embodiment, the loss value calculation subunit is used to calculate the respective loss value for each perturbation output sample by calling the target loss function or the black-box evaluation function.

[0106] In one embodiment, the disturbance processing subunit includes: The random sampling module is used to randomly sample a set of noise values ​​for each vector according to the noise scale. The noise addition and subtraction module is used to construct the original output plus noise and the original output minus the same noise respectively for each pair of positive and negative perturbations according to the dual sampling principle of the output stage.

[0107] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned output layer perturbation variance reduction likelihood ratio neural network training system 300 and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0108] The aforementioned output layer perturbation variance reduction likelihood ratio neural network training system 300 can be implemented as a computer program, which can be used in, for example... Figure 9 It runs on the computer device shown.

[0109] Please see Figure 9 , Figure 9 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0110] See Figure 9 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0111] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a neural network training method for reducing the likelihood ratio of the output layer perturbation variance.

[0112] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0113] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a neural network training method for reducing the likelihood ratio of the output layer perturbation variance.

[0114] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 9The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0115] The processor 502 is used to run a computer program 5032 stored in the memory to implement all the steps of the output layer perturbation variance reduction likelihood ratio neural network training method.

[0116] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0117] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0118] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform all steps of the output layer perturbation variance reduction likelihood ratio neural network training method.

[0119] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0120] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0121] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0122] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the system of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0123] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0124] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for training a neural network with reduced output layer perturbation variance and likelihood ratio, characterized in that, include: Obtain training data, neural network parameters, and output dimensions; set noise scale, number of samples, and batch size to obtain input data. The input data is subjected to a deterministic forward propagation computation using a neural network to obtain an initial output vector; The initial output vector is copied multiple times, and noise is introduced to form perturbed output samples. The corresponding loss value is obtained for each perturbed output sample. Based on the loss value and the perturbed output sample, the gradient signal of the output layer is constructed using the likelihood ratio formula; The gradient signal of the output layer is backpropagated to the network parameters through the vector-Jacobi product mechanism of the automatic differentiation framework, and the parameter gradient is calculated. The neural network parameters are then updated based on the parameter gradient.

2. The method for training a neural network with reduced output layer perturbation variance according to claim 1, characterized in that, The step of copying the initial output vector multiple times, introducing noise to form perturbed output samples, and obtaining the corresponding loss value for each perturbed output sample includes: The initial output vector is copied multiple times, noise is sampled according to the set noise scale, and the inverse noise is generated for half of the multiple vectors to obtain the perturbed output sample; Obtain the corresponding loss value for each perturbation output sample.

3. The method for training a neural network with reduced output layer perturbation variance according to claim 1, characterized in that, The step of obtaining the corresponding loss value for each perturbation output sample includes: For each perturbation output sample, the loss value is calculated by calling the target loss function or the black-box evaluation function.

4. The method for training a neural network with reduced output layer perturbation variance according to claim 1, characterized in that, The neural network includes CNN, ResNet, DenseNet, ViT, Transformer, BERT, Mamba, GAN, VAE, ACT, or other neural networks.

5. The method for training a neural network with reduced output layer perturbation variance according to claim 2, characterized in that, The perturbation output sample includes Gaussian noise or other symmetrical noise.

6. The method for training a neural network with reduced output layer perturbation variance according to claim 2, characterized in that, The process of copying the initial output vector multiple times, sampling noise according to a set noise scale, and generating the inverse noise for half of the multiple vectors to obtain perturbed output samples includes: For each vector, a set of noise values ​​is first randomly sampled according to the noise scale; According to the dual sampling principle of the output stage, if the set number of samples is even, then for each pair of positive and negative perturbations, the original output is constructed by adding noise and the original output is subtracted from the same noise to obtain the perturbation output sample.

7. A neural network training system for reducing the likelihood ratio of the output layer perturbation variance, characterized in that, include: The acquisition unit is used to acquire training data, neural network parameters, noise scale, number of samples, batch size, and output dimension to obtain input data; A forward propagation computation unit is used to perform a deterministic forward propagation computation on the input data using a neural network to obtain an initial output vector; The sample processing unit is used to copy the initial output vector multiple times, introduce noise to form perturbed output samples, and obtain the corresponding loss value for each perturbed output sample. A signal construction unit is used to construct the gradient signal of the output layer based on the loss value and the perturbed output sample using the likelihood ratio formula. The update unit is used to backpropagate the gradient signal of the output layer back to the network parameters through the vector-Jacobi product mechanism of the automatic differentiation framework, calculate the parameter gradient, and update the neural network parameters according to the parameter gradient.

8. A neural network training system for reducing output layer perturbation variance according to claim 7, characterized in that, The sample processing unit includes: The perturbation processing subunit is used to copy the initial output vector multiple times, sample noise according to the set noise scale, and generate the inverse noise of half of the multiple vectors to obtain the perturbation output sample. The loss value calculation subunit is used to obtain the corresponding loss value for each perturbation output sample.

9. A neural network training system for reducing output layer perturbation variance according to claim 8, characterized in that, The loss value calculation subunit is used to calculate the loss value for each perturbation output sample by calling the target loss function or the black-box evaluation function.

10. A neural network training system for reducing output layer perturbation variance according to claim 8, characterized in that, The disturbance processing subunit includes: The random sampling module is used to randomly sample a set of noise values ​​for each vector according to the noise scale. The noise addition / reduction module is used to construct the original output plus noise and the original output minus the same noise respectively for each pair of positive and negative perturbations, according to the dual sampling principle of the output stage. If the set number of samples is even, the perturbation output sample is obtained.

Images