Method and apparatus for sampling for back propagation in training of neural network model

Abstract

A computer-implemented method of sampling for back propagation in training of a neural network model having a plurality of layers, comprising: inputting a batch of training data into the neural network model; calculating a loss function based on an output of the batch of training data; a gradient of a trainable parameter of the neural network model is calculated based on the calculated loss function at least by calculating an activation gradient and a weight gradient for each layer of the neural network model, and wherein for each layer of the neural network model: the training data is sampled based on a first ratio, the first ratio is adjusted based on an activation random variance for calculating the activation gradient; and / or sampling the training data based on a second ratio for calculating the weight gradient, where the second ratio is adjusted based on a weight random variance; and updating the trainable parameters of the neural network model by the calculated gradient. Several other aspects are also disclosed.

Description

Technical Field

[0001] The various aspects of this disclosure generally relate to artificial intelligence, and more particularly to methods and apparatus for sampling backpropagation during the training of a neural network model. Background Technology

[0002] In recent years, neural networks have grown to significantly larger scales, leading to a substantial increase in the computational and time costs associated with training. To cite just a few examples, the computational costs required have increased exponentially with the emergence of Large Language Models (LLMs) or various transformer-based models. Therefore, reducing the computational complexity of neural networks and accelerating their training process while ensuring effectiveness has become a crucial issue.

[0003] Many efforts have been made to address this challenge. However, most of these efforts are tailored to specific contexts, such as data crawled from the web. This approach inevitably alters the original convergence trajectory. Modifications to any component of the training process (whether the model or the dataset) can lead to suboptimal behavior.

[0004] Therefore, a training method is needed that can reduce computational costs, accelerate the training process, and ensure effectiveness. Summary of the Invention

[0005] The following is a simplified overview of one or more aspects to provide a basic understanding of them. This overview is not a comprehensive summary of all anticipated aspects, and is neither intended to identify key or important elements of all aspects, nor to depict the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed descriptions that follow.

[0006] Gradient descent (GD) is one of the most commonly used methods for solving model parameters in machine learning algorithms, i.e., unconstrained optimization problems. When solving for the minimum of the loss function, gradient descent can be used to iteratively find the minimum loss function and the optimal model parameter values.

[0007] To accelerate training and reduce computational complexity, stochastic gradient descent (SGD) can be used instead of gradient descent (GD) by computing stochastic gradients instead of full gradients for backpropagation. However, due to the randomness of data selection, some structural variance may be introduced. As a common approach to SGD training, there are existing models... ,in It includes all trainable parameters and the dataset. and loss function In time You can choose the size as batch And the stochastic gradient is calculated as Meanwhile, the full gradient is calculated as Then the structural variance for: (1) in, Used to represent the sum of element-wise variances.

[0008] Furthermore, if some sampling methods are used to approximate the stochastic gradient, randomness variance will be introduced in addition to structural variance. Given a sampling distribution... The sampling method then obtains the sampled stochastic gradient. The variance of randomness relative to the sampling distribution for: (2)

[0009] Assuming the sampling is unbiased, i.e. Then the sampled stochastic gradient will still be a subset of the full gradient. The unbiased estimate, and the total variance It can be decoupled into and The simple addition is as follows: (3)

[0010] The method disclosed in this paper aims to reduce training costs while maintaining the integrity of the original convergence trajectory. Since the convergence speed of SGD is only negatively correlated with the gradient variance, in addition to the inherent convergence speed of batch gradient descent, a smaller variance is naturally preferred. In other words, if we ensure the randomness variance introduced by sampling... The structural variance is much smaller than that of SGD. Then we will get Under this variance control, similar gradients can be obtained for updates, and a similar convergence rate as the original SGD sampling method can be achieved, resulting in a similar convergence trajectory as the original SGD. Therefore, the problem has been decomposed into finding an unbiased sampling method for calculating gradients with the lowest possible variance.

[0011] This paper discloses a sampling method for calculating gradients, characterized by variance control. The disclosed method can sample activation gradients and weight gradients separately, thereby improving the efficiency of backpropagation and accelerating training.

[0012] In one aspect, a computer-implemented method is disclosed for sampling during backpropagation in training a neural network model having multiple layers. The computer-implemented method includes: inputting a batch of training data into the neural network model; calculating a loss function based on the output of the batch of training data; calculating gradients of trainable parameters of the neural network model based on the calculated loss function, at least by calculating activation gradients and weight gradients for each layer of the neural network model; and wherein, for each layer of the neural network model: sampling the training data based on a first ratio to calculate the activation gradient, wherein the first ratio is adjusted based on the activation random variance; and / or sampling the training data based on a second ratio to calculate the weight gradient, wherein the second ratio is adjusted based on the weight random variance; and updating the trainable parameters of the neural network model using the calculated gradients.

[0013] On the other hand, sampling the training data also includes sampling the training data in units of components, features, and / or data.

[0014] On the other hand, the first ratio is adjusted upwards based on the ratio of the activation random variance to the batch gradient variance being greater than a first threshold, or the first ratio is adjusted downwards based on the ratio of the activation random variance to the batch gradient variance being less than or equal to the first threshold.

[0015] On the other hand, the second ratio is adjusted upwards based on the ratio of the weight random variance to the batch gradient variance being greater than a second threshold, or the second ratio is adjusted downwards based on the ratio of the weight random variance to the batch gradient variance being less than or equal to the second threshold.

[0016] On the other hand, the first ratio and / or the second ratio are adjusted every certain number of training iterations.

[0017] On the other hand, the activation random variance, the weight random variance, or the batch gradient variance are obtained using the Monte Carlo method.

[0018] On the other hand, the batch gradient variance is obtained by running several iterations using different batches of training data without sampling to compute the activation gradient and the weight gradient; and obtaining the batch gradient variance with respect to the batch gradient corresponding to the number of iterations.

[0019] On the other hand, the activation random variance is obtained by: backpropagating multiple times for each iteration in a number of iterations, wherein only features are sampled to compute the activation gradient; compute the activation gradient for each backpropagation; compute the activation random variance for each iteration based on the computed activation gradient; and average the computed activation random variance over the number of iterations.

[0020] On the other hand, the weight random variance is obtained by: backpropagating multiple times for each iteration in a number of iterations; calculating the weight random variance for each backpropagation; and averaging all the calculated weight random variances over the number of iterations.

[0021] On the other hand, the first ratio is determined layer by layer, and the first ratio is monotonically non-decreasing as the layers of the neural network model become deeper.

[0022] On the other hand, for all layers, the first ratio is determined by the percentage of training data required to retain a specific proportion of the activation gradient norm.

[0023] On the other hand, the training data used to calculate the activation gradient is also sampled in proportion to one of the following: the gradient norm of each training data; the activation gradient norm of each training data before the final activation function; the gradient norm of each training data with respect to the parameters before the layer to be sampled; or the activation gradient norm of each training data of the output of the layer to be sampled.

[0024] On the other hand, the second ratio is determined for all layers of the neural network model.

[0025] On the other hand, the second ratio is determined relative to all trainable parameters of the neural network model.

[0026] On the other hand, the training data is sampled unbiasedly.

[0027] On the other hand, the Bernoulli distribution is used to sample the training data.

[0028] On the other hand, the neural network model is a transformer-based model.

[0029] On the other hand, the neural network model is a bidirectional encoder representation (BERT) from a transformer, and the input of a batch of training data into the neural network model further includes inputting a sequence of sentences into the BERT, wherein each sentence consists of several words.

[0030] In another aspect, the training data is sampled on a sentence-by-sentence basis to compute the activation gradient; and / or the training data is sampled on a label-by-label basis to compute the weight gradient.

[0031] On the other hand, the BERT is pre-trained for one or more of the following downstream tasks: lexical representation learning, text classification, named entity recognition, relation extraction, question answering system, or text generation.

[0032] In another aspect, the method further includes receiving input data by the trained BERT; and outputting corresponding results for the downstream task through the trained BERT.

[0033] In one aspect, a computer system is disclosed, comprising: one or more processors; and one or more storage devices storing computer-executable instructions that, when executed, cause the one or more processors to perform the operations of one or more methods herein.

[0034] In one aspect, one or more computer-readable storage media are disclosed that store computer-executable instructions, which, when executed, cause one or more processors to perform the operations of one or more methods herein.

[0035] In one aspect, a computer program product including computer-executable instructions is disclosed, which, when executed, cause one or more processors to perform the operations of one or more methods described herein. Attached Figure Description

[0036] The disclosed aspects will be described in conjunction with the accompanying drawings, which are provided for illustrative purposes and not for limiting the scope of the disclosure.

[0037] Figure 1 An exemplary neural network model 100 with multiple layers is shown according to various aspects of this disclosure.

[0038] Figure 2 An exemplary block diagram 200 for adaptive sampling for variance control of backpropagation according to various aspects of this disclosure is shown.

[0039] Figure 3 An exemplary flowchart 300 of adaptive sampling for variance control for backpropagation according to various aspects of this disclosure is shown.

[0040] Figure 4 An exemplary flowchart 400 is shown for using a trained neural network model for downstream tasks according to various aspects of this disclosure.

[0041] Figure 5 An exemplary computer system 500 according to various aspects of this disclosure is shown. Detailed Implementation

[0042] This disclosure will now be discussed with reference to several exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and thus implement the embodiments of this disclosure, and not to impose any limitation on the scope of this disclosure.

[0043] Various embodiments will be described in detail with reference to the accompanying drawings. Where possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. References to examples and embodiments are for illustrative purposes and are not intended to limit the scope of this disclosure. It should be noted that the word "based on" as used in this disclosure should be understood as "at least based on" and not "based on only" or "only based on".

[0044] The methods disclosed in the application may involve the use of user-related data. It should be noted that the use of user-related data requires user authorization and may not exceed the scope of user authorization.

[0045] Figure 1 An exemplary neural network model 100 with multiple layers is shown according to various aspects of this disclosure.

[0046] like Figure 1 As shown, a neural network model can include multiple layers, where the first layer is the input layer L1, and the last layer LN is the output layer. There are one or more intermediate layers, hidden layers, or blocks of multiple layers between the input and output layers. Figure 1 In this context, Ln represents the layer. Typically, a layer in a neural network model contains a linear component and an activation function that introduces a non-linear factor into the output.

[0047] As is known to those skilled in the art, the learning process of a neural network model typically includes forward propagation and backward propagation. In forward propagation, the input information I is fed into the input layer L1 and processed layer by layer before being transmitted to the output layer LN. If the expected output value cannot be achieved at the output layer LN, a loss function f is constructed based on the output value O and the ground truth, and then transformed into a backward propagation, also known as backpropagation. The partial derivatives of the loss function with respect to the trainable parameters are calculated layer by layer, forming the gradient of the loss function with respect to the trainable parameters. This serves as the basis for updating trainable parameters. The neural network model completes its learning process when the difference between the output value and the actual value reaches the expected value. Figure 1 Forward passes are shown in solid lines, and backward passes are shown in dashed lines.

[0048] Figure 1 This is shown as an example only, and other implementations are possible.

[0049] As neural networks grow to significantly larger scales, the dramatically increased computational and time costs of training have become a problem, with gradient computation accounting for a significant portion. To reduce computational complexity, especially for gradient computation, some current methods focus on approximating the forward pass, such as using the results of previous epochs as estimates or even Naive Bayes. These methods can inevitably alter the convergence trajectory. Therefore, in the method disclosed in this paper, the forward pass remains accurate to preserve the integrity of the original convergence curve, as it is indirect in gradient computation.

[0050] For backpropagation, which is directly related to gradients, this paper discloses the application of a sampling method to save computational effort. When using backpropagation to compute gradients, due to the chain rule, the computation is divided into two parts: activation gradients and weight gradients, corresponding to the intermediate and leaf nodes of the computation graph, or in other words, the gradients of the activation function part and the weight part of the layer in a neural network model. This paper discloses that activation gradients and weight gradients can be processed using their respective sampling rates.

[0051] Furthermore, similar to the structural variance and random variance Decoupling, It can be further decoupled into activating random variance and weighted random variance It is introduced by sampling, which is used to calculate the activation gradient and the weight gradient respectively, thus giving the total variance. As mentioned above, the convergence speed of SGD is only negatively correlated with the gradient variance. Therefore, the expected activation random variance generated by sampling the activation gradient and weight gradient is... and weighted random variance It will not exceed a specific threshold. This paper discloses a method that can be based on the activation random variance. and weighted random variance This is used to adjust the sampling rate of the activation gradient and the weight gradient.

[0052] Furthermore, while lower sampling rates result in greater speedup, convergence is more significantly affected. Additionally, as training progresses, gradients tend to become sparser across all dimensions, necessitating adjustments to the sampling rate based on different training stages. This paper discloses a method for adjusting the sampling rate of activation and weight gradients according to the training stage. Further details of the proposed sampling method for backpropagation are described below using examples.

[0053] Figure 2 An exemplary block diagram 200 for adaptive sampling for variance control in backpropagation according to various aspects of this disclosure is shown. Figure 2In the diagram, box 201 can represent the forward pass module; box 202 can represent the loss calculation module; box 203 can represent the backward pass module, where sub-box 203-1 can represent the activation sampling module, and sub-box 203-2 can represent the weight sampling module; box 204 can represent the parameter update module; and box 205 can represent the sampling rate determination module, where sub-box 205-1 can represent the activation sampling rate determination module, and sub-box 205-2 can represent the weight sampling rate determination module. Note that although all the boxes to be executed are described, this is for illustrative purposes only, and some boxes may be omitted in various embodiments.

[0054] In some examples, boxes 201-205 may be implemented in hardware that may include a processor and memory coupled to the processor, the memory being configured to perform one or more functions described herein (e.g., by the processor executing instructions stored in the memory). In some examples, boxes 201-205 may be implemented in a single piece of hardware; in other examples, boxes 201-205 or various combinations or components thereof may be implemented in discrete hardware that are not physically located together.

[0055] Alternatively or in some examples, boxes 201-205 may be implemented in code executed by a processor. If implemented in code executed by a processor, the functionality of boxes 201-205, or various combinations or components thereof, may be executed by a general-purpose processor, a special-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), or any combination of these or other programmable logic devices.

[0056] According to the embodiments disclosed herein, block 201 (forward propagation module) can support adaptive sampling for variance control used in backpropagation.

[0057] In one embodiment, block 201 receives training data, converts the training data into a suitable format for learning a neural network model with multiple layers, passes the converted training data forward layer by layer, and obtains output values ​​corresponding to the training data.

[0058] In one example, the neural network model could be a convolutional neural network (CNN), and the training data could be images, audio, sentences, or other suitable types of data.

[0059] In one example, the neural network model could be a transformer-based model, and the training data could be images or sentences, or other suitable types of sequence data.

[0060] In one example, the neural network model could be a bidirectional encoder representation (BERT) from a transformer, and the training data could be sentences or other suitable types of sequence data. Specifically, a sequence of sentences can be input into BERT, where each sentence consists of multiple words. The sentence sequence can be transformed into a sequence of tokens, where each word corresponds to a token, and where each sentence is separated by a special token [SEP]. The token sequence can then be encoded into an input embedding for subsequent forward passes, after which the output embedding is obtained. For example, let... , and Features are inputs, outputs, and weights, each with dimensions. , and ,in , , and This refers to the dimensions of the sentence, tags, input features, and output features. Therefore, in the forward pass performed in box 201, for the linear part of each layer, it will compute... And pass the activation function It will be input to the next level.

[0061] According to the embodiments disclosed herein, block 202 (loss calculation module) can support adaptive sampling for variance control used in backpropagation.

[0062] In one embodiment, block 202 receives output values ​​corresponding to training data from block 201 and calculates a loss that reflects the difference between the output values ​​and the expected values.

[0063] In one example, the expected value could be a ground truth label, as in supervised or semi-supervised learning.

[0064] In one example, the expected value can be a value inferred based on the features of the training data itself or contextual or temporal relationships, such as in unsupervised or semi-supervised learning.

[0065] According to the embodiments disclosed herein, block 203 (backpropagation module) can support adaptive sampling for variance control used in backpropagation.

[0066] In one embodiment, block 203 receives the calculated loss from block 202 and backward propagates the partial derivatives of the calculated loss with respect to the trainable parameters layer by layer to perform gradient calculation, and calculates the gradient with respect to the trainable parameters.

[0067] As mentioned above, when using backpropagation to compute gradients, due to the chain rule, the computation is divided into two parts: activation gradients and weight gradients, corresponding to the activation function part and weight part of a layer in a neural network model. Still using BERT as an example, in the backpropagation, it computes the activation gradient for each layer. and weight gradient .

[0068] In one embodiment, sub-frame 203-1 (activation sampling module) can support adaptive sampling for variance control for backpropagation according to embodiments disclosed herein.

[0069] In one example, subframe 203-1 samples the training data based on a first ratio to compute activation gradients in order to reduce computational complexity and speed up the training process. The first ratio indicates the percentage of training data that should be retained for computing activation gradients.

[0070] On one hand, the first rate used by sub-frame 203-1 is received from frame 205 (sampling rate determination block).

[0071] In some cases, different dimensions of the training data (such as those in BERT mentioned above) can be adjusted. , , and Sampling is then performed. For further steps, since the different components or features of the training data may not be independent in many scenarios, and the estimated activation gradients based on sampling will affect later computations, the variance introduced as backpropagation proceeds will further accumulate. Sampling in the data dimension rather than other dimensions such as components or features will be more suitable for this situation, as it behaves as if the gradient were being computed with some data discarded.

[0072] On the one hand, training data can be sampled in units of features, such as in the embedding dimension of BERT, other transformer-based models, or in RGB features of CNNs.

[0073] On the other hand, training data can be sampled in units of data components, such as by word or tag dimension in BERT, or by pixel in CNN.

[0074] On the other hand, training data can be sampled in units of data, such as by sentence dimension in BERT, or by image in CNN.

[0075] In other aspects, one or more of the above three aspects can be combined.

[0076] Then, given that not all samples are equal, sampling for more important data will yield better learning performance. Since the gradient reflects the direction in which the function changes most rapidly, it can be a suitable indicator of data importance. Furthermore, to control the stochastic variance of the published method, it has been shown that the optimal solution with the lowest variance for important sampling across the data dimensions is proportional to the gradient norm of each data point.

[0077] On the one hand, training data can be sampled in proportion to the gradient norm of the training data.

[0078] On the other hand, the training data can be sampled in proportion to the activation gradient norm of the training data before the final activation function.

[0079] On the other hand, training data can be sampled in proportion to the gradient norm of the parameters preceding the layer to be sampled.

[0080] On the other hand, training data can be sampled in proportion to the activation gradient norm of the training data of the output of the layer to be sampled.

[0081] On the one hand, training data can be sampled unbiasedly.

[0082] On the other hand, the training data can be sampled using the Bernoulli distribution, which is just an example, and other suitable distributions can be applied for unbiased sampling.

[0083] In one embodiment, according to the embodiments disclosed herein, subframe 203-2 (weighted sampling module) can support adaptive sampling for variance control used in backpropagation.

[0084] In one example, subframe 203-2 samples the training data based on a second ratio for calculating the weight gradients, in order to reduce computational complexity and speed up the training process. The second ratio indicates the percentage of training data that should be retained for calculating the weight gradients.

[0085] On one hand, the second ratio used by sub-frame 203-1 is received from frame 205 (sampling rate determination block).

[0086] Similar to the description of sampling activation gradients, sampling can also be performed on different dimensions of the training data for weight gradients. Using BERT as an example, this demonstrates how to compute weight gradients for a layer. There are three dimensions to matrix multiplication: input feature dimension. Output Dimensions Sentence multiplication by the tag dimension Sampling is possible in all dimensions.

[0087] On the one hand, training data can be sampled in units of features, such as in the embedding dimension of BERT, other transformer-based models, or in RGB features of CNNs.

[0088] On the other hand, training data can be sampled in units of data components, such as by word or tag dimension in BERT, or by pixel in CNN.

[0089] On the other hand, training data can be sampled on a data-by-data basis, for example, sampled at the sentence dimension in BERT, or sampled at the image dimension in CNN.

[0090] On the one hand, more than one dimension can be sampled simultaneously, because thorough sampling in more dimensions can have lower variance for the purpose of controlling the variance of randomness in the published method.

[0091] In other respects, it is possible to sample only one dimension, since how to divide the sampling rate between different dimensions that vary with different matrices may be a problem, and obtaining a solution in the future is computationally expensive.

[0092] On the other hand, for BERT, training data can be sampled only on a label-by-label basis, because the sentence is multiplied by the label dimension. Sampling in this context can take advantage of the inherent sparsity of sentences and words.

[0093] On the one hand, training data can be sampled unbiasedly.

[0094] On the other hand, the training data can be sampled using the Bernoulli distribution, which is just an example, and other suitable distributions can be applied for unbiased sampling.

[0095] In one aspect, the training data can be sampled in proportion to the norm of the gradient that is most informative.

[0096] On the other hand, leverage score sampling can be used to sample the training data.

[0097] In the example, the operations described using sub-boxes 203-1 and 203-2 can be performed individually to apply either activation gradient sampling or weight gradient sampling. In another example, the operations described using sub-boxes 203-1 and 203-2 can be performed together to apply both activation gradient sampling and weight gradient sampling.

[0098] According to the embodiments disclosed herein, block 204 (parameter update module) can support adaptive sampling for variance control used in backpropagation.

[0099] In one example, box 204 receives the calculated gradient relative to the trainable parameters and updates the trainable parameters using the calculated gradient.

[0100] In one example, box 204 sends updated parameters to boxes 201 and 203 to perform the next iteration using another batch of training data.

[0101] According to the embodiments disclosed herein, block 205 (sampling rate determination module) can support adaptive sampling for variance control used in backpropagation.

[0102] In one embodiment, block 205 sends the determined sampling rate to block 203, enabling block 203 to perform sampling.

[0103] In another embodiment, block 205 receives empirical data or calculations based on empirical data passed backward from block 203, and block 205 may determine the sampling rate based on the empirical data or its calculations.

[0104] As discussed earlier in this paper, a lower sampling rate will result in more acceleration, but convergence will be more significantly affected. Furthermore, the expected activation stochastic variance generated by sampling the activation and weight gradients... and weighted random variance It will not exceed a specific threshold. Furthermore, as training progresses, gradients tend to become sparser across several dimensions, necessitating adjustments to the sampling rate based on different training stages.

[0105] In this paper, the randomness variance introduced by sampling can be used as an indicator for adjusting the sampling rate. Threshold ratios much smaller than 1 are used as the basis for... and By imposing constraints to ensure that the randomness variance does not exceed the structural variance by too much, the convergence trajectory will hardly change. Furthermore, the sampling rate can be automatically adjusted using a threshold ratio, as a more aggressive sampling scheme can be tolerated when the gradient is sparser, using the same variance.

[0106] In one example, subbox 205-1 is based on activation random variance. and structural variance The ratio is used to determine the first ratio used to sample the training data to calculate the activation gradient, the structural variance. It can also be called batch gradient variance because it is calculated using a batch of training data instead of the entire set, compared to full gradient.

[0107] On the one hand, subframe 205-1 is based on activation random variance. and batch gradient variance The ratio is greater than a first threshold to determine the upward adjustment of the first ratio, and is based on the activation random variance. and batch gradient variance The ratio is determined by whether it is less than or equal to a first threshold. For example, the first threshold can be set to 0.01.

[0108] On the one hand, activate random variance and batch gradient variance It can be obtained using the Monte Carlo method.

[0109] On the other hand, in order to calculate the batch gradient variance This can be done through multiple iterations without sampling to compute the activation gradient and the weight gradient; this is also known as exact iteration. Batch gradient It can be calculated using empirical data from a single exact iteration, and the batch gradient variance Batch gradients from multiple exact iterations can be used for computation. For example, iterations can be performed by boxes 201-204, and empirical data from the iterations can be sent from box 203 to box 205, or computed... and It can be sent from box 203 to box 205.

[0110] On the other hand, in order to calculate the activation random variance It can run multiple iterations, and for each iteration, it runs multiple additional backpropagation iterations with different seeds and sampling only for the activation gradients. The activation gradients can be computed using empirical data from a single backpropagation iteration. Furthermore, the activation gradient variance for a single backpropagation iteration can be calculated using activation gradients from multiple backpropagation iterations, and then the average of the calculated activation gradient variances for multiple iterations can be obtained as... For example, iterations can be performed by boxes 201-204, and empirical data from the iterations can be sent from box 203 to box 205, or computed... and It can be sent from box 203 to box 205.

[0111] In a further aspect, it is used to obtain the batch gradient variance. and activation random variance Multiple iterations can be nested together to reduce computational complexity. For example, multiple exact iterations can be run, and for each exact iteration, multiple additional backpropagations with different seeds and sampling only the activation gradients can be run. Other combinations are possible.

[0112] Furthermore, layers closer to the output layer of the neural network model are more sensitive, and as backpropagation progresses, the norm of the activation gradients becomes increasingly sparse, aiming to gradually remove data with low gradient norms. Note that sampled data will not participate in the computation of shallower layers.

[0113] On the one hand, the first ratio used to sample the training data can be determined layer by layer and monotonically does not decrease as the layers deepen.

[0114] Furthermore, for all layers, it is expected that a certain percentage of the activation gradient norm will be retained, by... This indicates that the first ratio for each layer can be achieved based on the required percentage of data. The percentage of the total gradient norm is used to determine this.

[0115] On the one hand, for all layers, the first ratio can be determined by using the percentage of training data required to retain a specific proportion of the activation gradient norm.

[0116] On the other hand, based on activated random variance With batch gradient variance The ratio is greater than a first threshold to adjust the activation gradient norm of all layers upwards, and based on the activation random variance. With batch gradient variance The ratio of activation gradient norms across all layers is adjusted downwards by setting the ratio to be less than or equal to a first threshold. For example, the first threshold could be set to 0.01. Alternatively, a step size of 0.005 could be used to adjust the ratio of activation gradient norms across all layers.

[0117] In one example, subframe 205-2 is based on weighted random variance. and structural variance The ratio is used to determine the second ratio for sampling the training data to calculate the weight gradient. The structural variance can also be called the batch gradient variance because it is calculated using a batch of training data instead of the entire set, compared to the full gradient.

[0118] On the one hand, subframe 205-2 is based on weighted random variance. and batch gradient variance The ratio is greater than the second threshold to determine the upward adjustment of the second ratio, and is based on the weighted random variance. and batch gradient variance The ratio is determined by whether it is less than or equal to a second threshold. For example, the second threshold can be set to 0.01.

[0119] On the one hand, weighted random variance and batch gradient variance It can be obtained using the Monte Carlo method.

[0120] On the other hand, in order to calculate the batch gradient variance This can be done through multiple iterations without sampling to compute the activation gradient and the weight gradient; this is also known as exact iteration. Batch gradient It can be calculated using empirical data from a single exact iteration, and the batch gradient variance Batch gradients from multiple exact iterations can be used for computation. For example, multiple exact iterations can be performed via boxes 201-204, and empirical data from multiple exact iterations can be sent from box 203 to box 205, or computed... and It can be sent from box 203 to box 205.

[0121] On the other hand, in order to calculate the weighted random variance It can run multiple iterations, and the random variance of the weights can be theoretically calculated in each iteration, and then the average of the multiple iterations is used as the random variance of the weights. For example, when the label dimension of BERT is... When the sampling in the example is used, the random variance of the weights in the iteration can be calculated as follows: Other implementations are possible. For example, iterations can be performed via boxes 201-204, and empirical data from the iterations can be sent from box 203 to box 205, or computed... It can be sent from box 203 to box 205.

[0122] In a further aspect, the method used to obtain the aforementioned activation random variance and weighted random variance Multiple iterations can be nested together for execution to reduce computational complexity. For example, multiple iterations can be run, and for each iteration, another multiple backpropagations with different seeds and sampling only for activation gradients can be run, and the random variance of the weights can be computed in each activation sampling iteration. In another aspect, this is used to obtain the batch gradient variance. and activation random variance Multiple iterations can be nested together for execution, as described above. Other combinations are possible.

[0123] On the one hand, since the parameters have different sensitivities, a second ratio is determined for sampling the training data for all layers.

[0124] Furthermore, due to the ambiguous relationships between different layers in sampling, it may be difficult to obtain the optimal weight sampling rate that satisfies the second threshold. However, with slight scaling, it shifts to requiring sufficient conditions that maintain inequalities over all trainable parameters. Therefore, the second ratio can be determined on a parameter-by-parameter basis.

[0125] On the one hand, the second ratio is determined relative to all trainable parameters of the neural network model.

[0126] In one example, the operations described using sub-boxes 205-1 and 205-2 can be performed separately. In another example, the operations described using sub-boxes 205-1 and 205-2 can be performed together.

[0127] In one example, the operations described using subframes 205-1 and 205-2 can be performed every certain number of training iterations because the process required to obtain the variance would introduce some overhead as it requires additional iterations. However, as the weights gradually change during training, the distributions of the required variance and gradient norm also change slowly. Therefore, it is not necessary to perform sampling rate determination in every training iteration.

[0128] Figure 2 This is shown as an example only, and other implementations are possible.

[0129] Figure 3 An exemplary flowchart 300 of adaptive sampling for variance control for backpropagation according to various aspects of this disclosure is shown. As described below, some or all of the features shown may be omitted in implementations within the scope of this disclosure, and some of the shown features may not be required for implementations of all embodiments. Furthermore, some blocks may be executed in parallel or in a different order. In some examples, the method may be executed by any suitable means or unit for performing the functions or algorithms described below.

[0130] The method begins at box 301, where a batch of training data is fed into a neural network model, which includes multiple layers.

[0131] In one embodiment, the batch of training data is input and forward-passed through all layers of the neural network model.

[0132] In one embodiment, the neural network model may be a CNN, a transformer-based model, etc.

[0133] In one embodiment, the neural network model may be BERT, and the batch of training data may be a sequence of sentences, where each sentence consists of several words.

[0134] In one embodiment, the operation of block 301 can be performed by... Figure 2 The box 201 in the middle is executed.

[0135] The method then proceeds to box 302, where a loss function is calculated based on the output of the batch of training data.

[0136] In one embodiment, the operation of block 302 can be performed by Figure 2 Execute box 202 in the middle.

[0137] The method then proceeds to block 303, where the gradients of the trainable parameters of the neural network model are calculated based on the calculated loss function, at least by calculating the activation gradients and weight gradients for each layer of the neural network model.

[0138] In one embodiment, the operation of block 303 can be performed by Figure 2 Execute box 203 in the middle.

[0139] Additionally, in subframe 303-1, for each layer of the neural network model, the training data is sampled based on a first ratio to compute the activation gradient, wherein the first ratio is adjusted based on the activation random variance.

[0140] In one embodiment, training data can be sampled in units of components, features, and / or data. As an example, training data can be sampled in units of features, such as sampling along the embedding dimension in BERT, other transformer-based models, or sampling along RGB features in CNNs. As another example, training data can be sampled in units of data components, such as sampling along the word or tag dimension in BERT, or sampling along the pixel dimension in CNNs. As yet another example, training data can be sampled in units of data, such as sampling along the sentence dimension in BERT, or sampling along the image dimension in CNNs.

[0141] In one embodiment, the training data may be sampled proportionally to the gradient norm of the training data. In another embodiment, the training data may be sampled proportionally to the activation gradient norm of the training data prior to the final activation function. In yet another embodiment, the training data may be sampled proportionally to the gradient norm of the training data with respect to the parameters prior to the layer to be sampled. In still another embodiment, the training data may be sampled proportionally to the activation gradient norm of the training data of the output of the layer to be sampled.

[0142] In one embodiment, the training data can be sampled unbiasedly. In another embodiment, the training data can be sampled using a Bernoulli distribution; this is merely an example, and other suitable distributions for unbiased sampling can be applied.

[0143] In one embodiment, the operation of sub-block 303-1 can be performed by Figure 2 Execute subframe 203-1 within the scope.

[0144] Additionally, in subframe 303-2, for each layer of the neural network model, the training data is sampled based on a second ratio to compute the weight gradient, where the second ratio is adjusted based on the weight random variance.

[0145] In one embodiment, training data can be sampled in units of components, features, and / or data. As an example, training data can be sampled in units of features, such as sampling along the embedding dimension in BERT, other transformer-based models, or sampling along RGB features in CNNs. As another example, training data can be sampled in units of data components, such as sampling along the word or tag dimension in BERT, or sampling along the pixel dimension in CNNs. As yet another example, training data can be sampled in units of data, such as sampling along the sentence dimension in BERT, or sampling along the image dimension in CNNs.

[0146] In another embodiment, more than one dimension can be sampled simultaneously.

[0147] Alternatively, sampling may be performed on only one dimension. In another embodiment, for BERT, training data may be sampled only on a per-label basis in the sentence-multiplied-label dimension.

[0148] In one embodiment, the training data can be sampled unbiasedly.

[0149] In another embodiment, the training data can be sampled using a Bernoulli distribution; this is merely an example, and other suitable distributions for unbiased sampling can be applied.

[0150] In one embodiment, the training data can be sampled in proportion to the norm of the most informative gradient.

[0151] In another implementation, the training data can be sampled using a leverage score sampling method.

[0152] In one embodiment, the operation of sub-block 303-2 can be performed by Figure 2 Execute subframe 203-2 within the scope.

[0153] In one embodiment, the methods described using sub-blocks 303-1 and 303-2 can be executed separately, or the methods described using sub-blocks 203-1 and 203-2 can be executed together.

[0154] The method then proceeds to box 304, where the trainable parameters of the neural network model are updated by the calculated gradients.

[0155] In one embodiment, the operation of block 304 can be performed by Figure 2The box 204 in the middle is executed.

[0156] In one embodiment, the operations of blocks 301-304 can be performed iteratively until convergence or the expected output value is reached.

[0157] Additionally, before proceeding to box 303, the method moves to box 305, where the first ratio used for activating gradient sampling and / or the second ratio used for weight gradient sampling are adjusted.

[0158] Additionally, in sub-box 305-1, the first ratio is adjusted based on the activation random variance in the following ways: the first ratio is adjusted upwards based on the ratio of the activation random variance to the batch gradient variance being greater than a first threshold; or the first ratio is adjusted downwards based on the ratio of the activation random variance to the batch gradient variance being less than or equal to a first threshold. For example, the first threshold can be set to 0.01.

[0159] In one embodiment, the activation random variance and / or batch gradient variance can be obtained using the Monte Carlo method.

[0160] In one embodiment, the batch gradient variance can be obtained by running several iterations using different batches of training data, without sampling to compute the activation gradient and the weight gradient, and the batch gradient variance relative to the batch gradient corresponding to the number of iterations.

[0161] In one embodiment, the activation random variance can be obtained by: backpropagating multiple times for each iteration in a number of iterations, wherein only features are sampled to compute the activation gradient; compute the activation gradient for each backpropagation; compute the activation random variance for each iteration based on the computed activation gradient; and average the computed activation random variance over the number of iterations.

[0162] In the example, multiple iterations for obtaining the batch gradient variance and activation random variance can be nested together; that is, multiple exact iterations can be run, and for each exact iteration, additional backpropagations with different seeds and sampling only for activation gradients can be run.

[0163] In one embodiment, the first ratio used to sample the training data can be determined layer by layer and monotonically does not decrease as the layer deepens.

[0164] In one embodiment, for all layers, the first ratio may be determined by the percentage of training data required to retain a specific proportion of the activation gradient norm.

[0165] In another embodiment, the activation gradient norm ratio can be adjusted upward for all layers based on the ratio of activation random variance to batch gradient variance being greater than a first threshold, and the activation gradient norm ratio can be adjusted downward for all layers based on the ratio of activation random variance to batch gradient variance being less than or equal to the first threshold.

[0166] In one embodiment, the operation of sub-block 305-1 can be performed by Figure 2 Execute subframe 205-1 within the scope.

[0167] Additionally, in sub-box 305-1, the second ratio can be adjusted based on the weighted random variance in the following ways: by adjusting the second ratio upwards based on the ratio of the weighted random variance to the batch gradient variance being greater than a second threshold, or by adjusting the second ratio downwards based on the ratio of the weighted random variance to the batch gradient variance being less than or equal to a second threshold. For example, the second threshold can be set to 0.01.

[0168] In one embodiment, the weight random variance and the batch gradient variance can be obtained using the Monte Carlo method.

[0169] In one implementation, the batch gradient variance can be obtained by running several iterations using different batches of training data, without sampling to compute activation gradients and weight gradients, and the batch gradient variance relative to the batch gradient corresponding to the number of iterations.

[0170] In one embodiment, the weight random variance can be obtained by: backpropagating multiple times for each iteration in a number of iterations; calculating the weight random variance for each backpropagation; and averaging all the calculated weight random variances over the number of iterations.

[0171] In one example, when BERT's label dimension When the sampling in the example is used, the random variance of the weights in the iteration can be calculated as follows: Other implementations are possible.

[0172] In one example, the multiple iterations described above for obtaining the random variance of activation and the random variance of weights can be nested together for execution; that is, multiple iterations can be run, and for each iteration, additional backpropagations with different seeds and sampling only for activation gradients can be run.

[0173] In another example, multiple iterations for obtaining the batch gradient variance and the activation random variance can be nested together to be performed, as described above.

[0174] In one embodiment, the second ratio can be determined for all layers of the neural network model.

[0175] In one embodiment, the second ratio can be determined separately with respect to all trainable parameters of the neural network model.

[0176] In one embodiment, the operation of sub-frame 305-2 can be performed by... Figure 2 Execute subframe 205-2 within the scope.

[0177] In one embodiment, the operations of subframes 305-1 and 305-2 can be performed every certain number of training iterations.

[0178] In one embodiment, the operations of sub-frames 305-1 and 305-2 can be performed separately or in combination.

[0179] Figure 3 This is shown as an example only, and other implementations are possible.

[0180] Table 1 shows an example implementation of BERT, which combines the above-mentioned features. Figure 3 The embodiments described in box 303-1 are merely examples and other implementations are possible.

[0181] In one embodiment, the method shown in Table 1 can be derived from... Figure 2 Execute boxes 201, 202, 204 and sub-box 203-1. Table 1

[0182] Table 2 shows example implementations of BERT, which combine several embodiments described above for box 303-2. This is merely an example and other implementations are possible.

[0183] In one embodiment, the method shown in Table 2 can be performed by... Figure 2 Execute boxes 201, 202, 204 and sub-box 203-2. Table 2

[0184] Table 3 shows example implementations combining several embodiments described above for blocks 305-1 and 305-2. These are merely examples and other implementations are possible.

[0185] In one embodiment, the method shown in Table 3 can be derived from... Figure 2 Execute boxes 201-205 in the middle. Table 3

[0186] Figure 4An exemplary flowchart 400 for using a trained neural network model for a downstream task according to various aspects of this disclosure is shown. As described below, some or all of the features shown may be omitted in implementations within the scope of this disclosure, and some of the shown features may not be required for all embodiments. Furthermore, some blocks may be executed in parallel or in a different order. In some examples, the method may be performed by any suitable means or unit for performing the functions or algorithms described below.

[0187] The method begins at box 401, where input data is received by a trained neural network.

[0188] In one embodiment, the trained neural network is BERT, and it is pre-trained for one or more of the following downstream tasks: lexical representation learning, text classification, named entity recognition, relation extraction, question answering system, or text generation.

[0189] In one embodiment, the input data may be raw text, words, phrases, sentences, paragraphs, or other suitable text formats.

[0190] In one embodiment, input data can be received remotely via a communication link. For example, BERT can be deployed as part of the backbone of a search engine, Q / A system, etc.

[0191] Then, the method proceeds to box 402, where the trained BERT is used to output the corresponding results for the downstream task.

[0192] For example, in a lexical representation learning task, the output could be word or phrase embeddings, which can be used as input features for other natural language processing tasks. As another example, in a text classification task, the output could be labels used for classification. For yet another example, in a named entity recognition task, the output could be extracted entities with specific meanings. For yet another example, in a relation extraction task, the output could be extracted relations between entities. For yet another example, in a question-answering system task, the output could be the answer to the input question. And for yet another example, in a text generation task, the output could be coherent and semantically plausible text generated by BERT.

[0193] Figure 5 An exemplary computer system 500 according to various aspects of this disclosure is illustrated. The computer system may include at least one processor 510. The computer system may also include at least one storage device 520. It should be understood that the storage device 520 may store computer-executable instructions that, when executed, cause the processor 510 to perform actions as described above. Figure 1-4 Any operation described in accordance with the embodiments of this disclosure.

[0194] Embodiments of this disclosure may be embodied in one or more computer-readable media, such as non-transitory computer-readable media. A non-transitory computer-readable medium may store computer-executable instructions that, when executed, cause one or more processors to perform actions as described above. Figure 1-4 Any operation described according to embodiments of this disclosure.

[0195] Embodiments of this disclosure can be embodied in a computer program product including computer-executable instructions that, when executed, cause one or more processors to perform according to the combination Figure 1-4 Any operation of the embodiments described in this disclosure.

[0196] It should be understood that all operations in the above methods are merely exemplary, and this disclosure is not limited to any operation in the methods or the order of such operations, and should cover all other equivalents under the same or similar concepts.

[0197] It should also be understood that all modules in the aforementioned apparatus can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. Furthermore, any of these modules can be further functionally divided into sub-modules or combined together.

[0198] The foregoing description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to limit them to the aspects shown herein. All structural and functional equivalents of elements throughout the various aspects described in this disclosure that are known or will be known hereafter by those skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims.

Claims

1. A computer-implemented method for sampling backpropagation during training of a neural network model with multiple layers, comprising: A batch of training data is input into the neural network model; The loss function is calculated based on the output of the training data from the batch. The gradients of the trainable parameters of the neural network model are calculated based on the calculated loss function, at least by computing the activation gradients and weight gradients for each layer of the neural network model, and wherein, for each layer of the neural network model: The training data is sampled based on a first ratio to calculate the activation gradient, wherein the first ratio is adjusted based on the activation random variance; and / or The training data is sampled based on a second ratio to calculate the weight gradient, wherein the second ratio is adjusted based on the weight random variance; and The trainable parameters of the neural network model are updated using the calculated gradients.

2. The computer-implemented method according to claim 1, wherein, Sampling the training data further includes: The training data is sampled in units of components, features, and / or data.

3. The computer-implemented method according to claim 1, wherein, The first ratio is adjusted based on the activated random variance in the following manner: The first ratio is adjusted upwards based on the ratio of the activation random variance to the batch gradient variance being greater than a first threshold, or downwards based on the ratio of the activation random variance to the batch gradient variance being less than or equal to the first threshold.

4. The computer-implemented method according to claim 1, wherein, The second ratio is adjusted based on the weighted random variance in the following manner: The second ratio is adjusted upwards based on the ratio of the weight random variance to the batch gradient variance being greater than a second threshold, or downwards based on the ratio of the weight random variance to the batch gradient variance being less than or equal to the second threshold.

5. The computer-implemented method according to claim 3 or 4, wherein, The first ratio and / or the second ratio are adjusted every certain number of training iterations.

6. The computer-implemented method according to claim 3 or 4, wherein, The activation random variance, the weight random variance, or the batch gradient variance are obtained using the Monte Carlo method.

7. The computer-implemented method according to claim 3 or 4, wherein, The batch gradient variance is obtained in the following way: Run several iterations using different batches of training data without sampling to compute the activation gradient and the weight gradient; The batch gradient variance is obtained with respect to the batch gradient corresponding to the number of iterations.

8. The computer-implemented method according to claim 3, wherein, The activation random variance is obtained in the following way: The iteration is backward propagated multiple times for each iteration in several iterations, wherein only features are sampled to calculate the activation gradient; Calculate the activation gradient for each backpropagation; The activation stochastic variance for each iteration is calculated based on the calculated activation gradient. The calculated activation random variance is averaged over the number of iterations.

9. The computer-implemented method according to claim 4, wherein, The weighted random variance is obtained in the following way: For each iteration in several iterations, the backward propagation is performed multiple times; Calculate the random variance of the weights for each backward pass; The random variance of all calculated weights is averaged for the number of iterations.

10. The computer-implemented method according to claim 1, wherein, The first ratio is determined layer by layer, and the first ratio is monotonically non-decreasing as the neural network model becomes deeper.

11. The computer-implemented method according to claim 10, wherein, For all layers, the first ratio is determined by the percentage of training data required to retain a specific proportion of the activation gradient norm.

12. The computer-implemented method according to claim 1, wherein, The training data used to calculate the activation gradient is also sampled in proportion to one of the following: Gradient norm of each training data point; The activation gradient norm for each training data point before the final activation function; The gradient norm of each training data point with respect to the parameters preceding the layer to be sampled; or The activation gradient norm of each training data in the output of the layer to be sampled.

13. The computer-implemented method according to claim 1, wherein, The second ratio is determined for all layers of the neural network model.

14. The computer-implemented method according to claim 13, wherein, The second ratio is determined relative to all trainable parameters of the neural network model.

15. The computer-implemented method according to claim 1, wherein, The training data was sampled unbiasedly.

16. The computer-implemented method according to claim 15, wherein, The training data is sampled using a Bernoulli distribution.

17. The computer-implemented method according to claim 1, wherein, The neural network model is a transformer-based model.

18. The computer-implemented method according to claim 1, wherein, The neural network model is a bidirectional encoder representation (BERT) derived from the transformer, and The process of inputting a batch of training data into the neural network model also includes inputting sentence sequences into the BERT, where each sentence consists of several words.

19. The computer-implemented method according to claim 18, wherein, The training data is sampled sentence by sentence to calculate the activation gradient; and / or The training data is sampled in units of labels to calculate the weight gradient.

20. The computer-implemented method according to claim 18, wherein, The BERT is pre-trained for one or more of the following downstream tasks: lexical representation learning, text classification, named entity recognition, relation extraction, question answering system, or text generation.

21. The computer-implemented method according to claim 20, further comprising: The trained BERT receives the input data; as well as The trained BERT is used to output the corresponding results for the downstream task.

22. A computer system, comprising: One or more processors; as well as One or more storage devices storing computer-executable instructions, which, when executed, cause the one or more processors to perform the operations of the method as described in any one of claims 1-21.

23. One or more computer-readable storage media storing computer-executable instructions, which, when executed, cause one or more processors to perform the operations described in any one of claims 1-21.

24. A computer program product comprising computer-executable instructions, which, when executed, cause one or more processors to perform the operations described in any one of claims 1-21.

Images