Machine learning systems that enable effective training

Abstract

The machine learning system implements a machine learning model. The system includes at least one layer of processing nodes, each processing node including a processor that executes computer-readable instructions to perform at least one operation based on one or more inputs received at the processing node. The operations are scaled by a first scaling factor calculated so that the variance of an output of the at least one operation has a target variance, e.g., unit variance or a variance that matches the variance of the inputs.

Description

[Technical Field]

[0001] The present disclosure relates to data processing in machine learning computers. [Background technology]

[0002] The development of algorithms that efficiently utilize available hardware has been key to the significant advances seen in deep learning over the past decade. As state-of-the-art models grow in size, hardware efficiency is also motivated by the need to lower the cost of training. These costs have become prohibitive in terms of money, time, and environmental impact. However, with the end of Moore's Law and Dennard scaling, improvements in transistor density can no longer be relied upon to provide a simple path to efficiency improvements, and other technologies must be leveraged.

[0003] One such technique is the use of low-precision numeric formats. The gains from this technique are significant, with computation, memory, and bandwidth usage all dependent on the bit-width of the format. Recently, mixed-precision training has been developed, which allows for the use of different numeric formats for various activations, weights, and gradients (collectively known as tensors) in the training process. See, for example, "Mixed-Precision Training" by Micikevicius, Paulius, et al., International Conference on Representation Learning, 2018. In such schemes, there are often efficiency benefits to using low-precision formats for as many tensors as possible.

[0004] Low-precision formats must trade off the range of representable values ​​for precision (corresponding to the interval between the represented values). In IEEE-754-based floating-point formats, this is controlled by the number of bits in the format allocated to the exponent versus the mantissa. This tradeoff is illustrated in Figure 1, which shows the SNR (signal-to-noise ratio) of normally distributed samples quantized in FP16 and FP8. Two 8-bit formats, "FP8E4" and "FP8E5", allocate 4 and 5 exponent bits, respectively (1 bit for the sign, 3 or 2 bits for the mantissa). E5 provides a wider range at the expense of precision, while E4 provides a narrower range while increasing precision.

[0005] The limited range and precision of numeric formats introduce two forms of errors: clipping errors, which occur when values ​​are outside the representable range, and quantization errors, which occur when values ​​are between representable numbers. Both errors can degrade the deep learning training process. Therefore, techniques that make the deep learning training process more robust to range reduction and quantization errors are essential for efficient training in low-precision formats.

[0006] A number of approaches are described below.

[0007] Loss scaling—The range reduction in FP16 and FP8 is particularly challenging for the backward pass (i.e., backpropagation where weights are adjusted), and standard model design practices result in gradients that risk underflow. To address this, Micikevicius et al. (2018) observed that the loss can be multiplied by a scalar to increase the gradient scale, and the weight gradients are then divided by the same scalar in the optimizer. This works due to the linearity of the backward pass implicit in the chain rule. Loss scaling is essential for high- and mixed-precision training in FP16 and FP8. However, the choice of loss scale is not theoretically motivated and must be found empirically. This has several drawbacks. First, hyperparameter sweeps must be performed to find the loss scale value. Insufficient loss scale may only become apparent later in training, requiring multiple runs. Second, it is not clear in advance what changes require a re-sweep of the loss scale. Third, because loss scaling only applies a single global scaling factor, it has no mechanism to deal with scale differences between gradient tensors, which can be too large for effective learning in some models.

[0008] Automatic Loss Scaling - Dynamic adjustment of the loss scale during training is called automatic loss scaling (Kuchaiev, Oleksii, et al., "Mixed-Precision Training for Natural Language Processing (NLP) and Speech Recognition with OpenSeq2Seq," arXiv preprint arXiv:1805.10387 (2018)). This eliminates the need to sweep the initial loss scale and counters shifts in tensor distributions during training. Dynamic schemes require gradient overflow detection or tensor statistics collection as a basis for changing the scale. Updates containing overflowed values ​​may have to be discarded, and such schemes cannot scale across tensors.

[0009] Tensor-wise scaling—To address the scaling challenges inherent in FP8 training, Micikevicius, Paulius, et al., "FP8 Format for Deep Learning," arXiv preprint (arXiv:2209.05433, 2022), proposed a tensor-wise scaling method that locally rescales based on runtime statistics. This technique allows for appropriately scaled tensors across the entire model. However, recording statistics for multiple tensors can incur additional computation, memory, bandwidth, and inter-device communication costs. Furthermore, strategies for adjusting scaling factors may require tuning hyperparameters, potentially increasing implementation complexity. Summary of the Invention

[0010] The present disclosure addresses certain technical problems, particularly the technical problem of being able to effectively initialize or train machine learning models using low-precision number formats and mixed-precision number formats (mixed precision includes low-precision number formats), such as FP16 and FP8.

[0011] The present disclosure relates to a machine learning system including a hardware computer configured to execute instructions on a processor including one or more processing units. The instructions may be stored in a memory accessible to the processor. The present disclosure further relates to a method for generating a computer program for implementing a machine learning model for execution on the machine learning system. The machine learning model may be a neural network. The present disclosure addresses a specific technical problem.

[0012] The method and system allow for the implementation of machine learning models that can be trained with improved precision and accuracy compared to existing machine learning models of the same numeric format.

[0013] According to one aspect of the present disclosure, there is provided a machine learning system that implements a machine learning model, the system comprising at least one tier of processing nodes, each processing node including a processor configured to execute computer-readable instructions to perform at least one operation based on one or more inputs received at the processing node, the at least one operation implementing the machine learning model being scaled by a first scaling factor calculated such that the variance of an output of the at least one operation has a target variance.

[0014] According to another aspect of the present disclosure, there is provided a computer-implemented method including: receiving a computation graph including a plurality of nodes, each node of the plurality of nodes corresponding to a computational operation for training a machine learning model, and a plurality of edges, each edge connecting a pair of the nodes and corresponding to an output of a first node of the pair of nodes and an input to a second node of the pair of nodes; inserting a first scaling factor into the computation graph associated with at least one of the plurality of nodes, the first scaling factor being calculated such that a variance of an output of the at least one node has a target variance.

[0015] According to another aspect of the present disclosure, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium including computer-executable instructions that, when executed, implement a neural network, the instructions including first code embodying at least one scaling operation configured to receive a tensor of weights and a tensor of input activations and to generate a tensor of output activations having a target variance. [Brief explanation of the drawings]

[0016] For a better understanding of the invention and to show examples thereof, reference will now be made to the accompanying drawings, which are given by way of example only and do not exclude other embodiments of the invention, in which: [Figure 1] FIG. 1 is a graph illustrating the signal-to-noise ratio associated with a low precision numeric format. [Figure 2A] FIG. 2A is a schematic diagram illustrating scaling of feedforward network layers in an example machine learning system. [Figure 2B] FIG. 2B is a histogram of exponent values ​​at initialization of the feedforward network of FIG. 2A. [Figure 3] FIG. 3 is a table illustrating example unit scaling factors for use in an exemplary machine learning system. [Figure 4] FIG. 4 is a schematic block diagram of a first example computer system. [Figure 5] FIG. 5 is a schematic block diagram of a second example computer system. [Figure 6] FIG. 6 is a schematic block diagram of a third example computer system. [Figure 7] Figure 7 is a table comparing techniques for low-precision training of machine learning models. [Figure 8] Figure 8 is a series of code snippets illustrating an implementation of an exemplary technique using PyTorch. [Figure 9] FIG. 9 is a series of graphs showing the performance of various models trained using different scaling techniques on a written language modeling task. [Figure 10] FIG. 10 is a table showing the performance of a system employing an exemplary technique on a masked language modeling task. [Figure 11] Figure 11 is a table showing common floating-point representations in deep learning. [Figure 12] Figure 12 is a code snippet showing the implementation of the unit scaling model in PyTorch. [Figure 13]Figure 13 is a series of graphs comparing different residual scaling approaches on a written language modeling task. [Figure 14] Figure 14 is a table showing the hyperparameters used in the character modeling task. [Figure 15] FIG. 15 is a table showing the results of the character language modeling task using different models and accuracies. [Figure 16] Figure 16 is a table showing the hyperparameters used in the masked language modeling task. [Figure 17] Figure 17 shows a histogram of absolute tensor values ​​for a model performing masked language modeling using loss scaling at the start of training. [Figure 18] Figure 18 is a histogram of absolute tensor values ​​in a model performing masked language modeling using unit scaling at the start of training. [Figure 19] Figure 19 shows a histogram of absolute tensor values ​​for a model performing masked language modeling with loss scaling at the end of training. [Figure 20] Figure 20 is a histogram of absolute tensor values ​​for a model performing masked language modeling using unit scaling at the end of training. [Figure 21] FIG. 21 is a schematic block diagram of an exemplary computing system. DETAILED DESCRIPTION OF THE INVENTION

[0017] Unit scaling is a paradigm for designing deep learning models that simplifies the use of lower-precision numerical formats. Training in FP16 or the recently proposed FP8 format offers significant efficiency gains but may lack sufficient range for out-of-the-box training. Unit scaling addresses this issue by introducing a principled approach to model numerical computations that seeks unit variance for all weights, activations, and gradients at initialization. Unlike other methods, this approach does not require multiple training runs to find an appropriate scale, nor does it incur significant computational overhead. It works for a variety of models and optimizers, allowing immediate training in FP16 or FP8 without loss of accuracy. This disclosure also provides a procedure for adapting existing models to unit scale. Unit scaling can be extended to other target scales.

[0018] Identity scaling addresses the range reduction problem identified above by attempting to place the model's tensors within their representable range at initialization.

[0019] For normally distributed tensors, the term "scale" refers to the standard deviation. The mean (over the range of the format) has minimal variation. Scale characterizes the probability of clipping errors occurring given the format. Over- or under-scale can cause values ​​to deviate outside the representable range. The ability to predict the scale of a tensor in a deep learning model would be a powerful tool for addressing clipping errors. While this is generally difficult, the problem becomes simple during initialization. Before the training step, parameters are drawn from a known initialization distribution, so the scale of each tensor can be derived through analysis or simulation, given the input distribution. A further simplification is to make local distribution assumptions for one layer in the model and consider the propagation of scale through the model. This allows for a methodological analysis: first, to evaluate the scaling effect of each operation independently; and second, to propagate scale back and forth through the computational graph.

[0020] Because the initial distributions of parameters are directly controlled by the model designer, the dominant approach to scaling is to choose initial parameter distributions to trade off forward-pass variance scaling and backward-pass variance scaling. (Xavier Glorot and Yoshua Bengio, Understanding the Difficulty of Training Deep Feedforward Neural Networks, 13th International Conference on Artificial Intelligence and Statistics, 2010; Kaiming He et al., Deep Dive into Rectifying Elements: Surpassing Human-Level Performance in ImageNet Classification, IEEE International Conference on Computer Vision, 2015.) Such schemes were developed to avoid exploding / vanishing gradients in deep multilayer perceptrons. As such, they do not attempt to constrain the scale of parameters or parameter gradients. They are also limited to computations where scaling factors can be moved to learnable parameters.

[0021] Identity scaling uses a similar scale analysis technique, but instead of modifying the initialization scale of parameter tensors, it inserts a scaling factor into the computation graph. This helps control the scale of intermediate tensors, giving a more general approach than initialization-based schemes.

[0022] Unit scaling is a technique for building deep learning models based on graph construction recipes that insert scaling factors into the computational graph that describes the training or inference process. As implied by the name "unit scaling," the default version of the recipe aims to achieve near-unit scale (i.e., standard deviation = 1) for internal tensors and parameters upon initialization. However, the recipe can be generalized to any target scale and is not necessarily limited to "unit" scaling.

[0023] This is achieved by inserting scaling factors in the forward and backward passes. This is illustrated in Figure 2A, which shows that each tensor in the feedforward network (FFN) layer 10 of the example machine learning system is multiplied by a fixed scalar to achieve a consistent scale. In particular, the dotted rectangle 11 indicates the fixed scalar applied to the tensor. For clarity of the illustration, not all dotted rectangles are labeled. The tensors shown in Figure 2 are as follows: x1 represents the input tensor, w1 represents the input weight tensor corresponding to the input tensor, x2 represents the matrix multiplication of the weight w1 and the input x1, x3 represents the output activation of activation function 12 of layer 10, which in this case is a GeLU function, w2 represents the output weight tensor corresponding to the output activation, x4 represents the matrix multiplication of the output activations and output weights, and the same labels starting with ∇ represent the corresponding gradient tensors.

[0024] The solid rectangle 13 in Figure 2A represents the application of scaling in the loss scaling technique. That is, the solid rectangle represents an alternative to the unit scaling technique described herein. Figure 2B shows the exponent values ​​of the above tensor upon initialization of the FFN layer 10. The upper histogram uses loss scaling. The lower histogram uses unit scaling as described herein. The shading of the histogram represents the bin density. The Y-axis reflects the exponent values ​​available in FP16, and the dashed lines indicate the maximum / minimum exponents in the FP8E4 format.

[0025] Like loss scaling, backward pass corrections also ensure correct gradients up to a certain scaling factor. However, unlike loss scaling, unit scaling determines these scales based on a fixed set of rules for each operation, rather than a single hyperparameter found empirically or via an adaptive algorithm. The selected scales allow each operation to approximately preserve the variance of its inputs. This effect propagates throughout the model, giving global unit scaling. By concentrating values ​​approximately in the center of the exponent range during initialization, tensors are given headroom to potentially shift during training without going out of range. It should be noted that unit scaling can be used in conjunction with a system that monitors or adapts specific scaling factors during training.

[0026] It will be appreciated that this is just one example of a suitable neural network layer to which the concepts herein may be applied. As noted above, unit scaling can be applied to nearly all operations, including different types of layers (e.g., attention layers) performed in neural networks.

[0027] Inserting scaling factors into a computation graph may involve storing the scaling factors as attributes of existing operations, such that each scaling factor is associated with the relevant operation in the graph. In other words, each node / operation in the graph consists of multiple attributes, one of which is the scaling factor.

[0028] Alternatively, an additional operation may be inserted into the computation graph by splitting an edge in the graph into two edges connected by a scaling operation node, which has a single input and a single output and multiplies all elements of the input tensor by the same fixed scale.

[0029] definition A "deep learning model" is a differentiable function from inputs and learnable parameters to outputs. A computational graph is a graph of operations (nodes) and tensors (edges) that describe the structure of a computation. Tensors represent the inputs to and outputs from nodes. Typically, there is one "forward graph" that implements the model (the mapping from inputs and learnable parameters to outputs). There are one or more "backward graphs" that implement the gradients (e.g., mapping from inputs to parameter gradients). An "operation" is a mapping from input tensors to output tensors. Many operations are differentiable to produce gradient operations. A "scaling operation" is an operation that has a forward scaling parameter that is multiplied by the output, and a backward scaling parameter that is multiplied by the result of the gradient operation for each input tensor. A "scaling factor" is a scalar value that is multiplied by a tensor to change its scale. "Parameters" are tensors that have initial values ​​(usually chosen randomly) and are updated during training (usually using a first-order optimizer based on the gradient of the loss). The "bias" parameter is additive and is usually initialized to zero. A "cut edge" is an edge in the forward graph that, when cut, disconnects the head and tail of the graph at that edge (i.e., there is no other path from the head to the tail of the forward graph).

[0030] The high level recipe (i.e., method) for unit scaling disclosed herein is as follows: 1. Initialize the unbiased parameters with unit variance and whiten the input. 2. Calculate the scaling factors for all (scaled) operations. 3. Identify a non-cut edge and constrain the operation consuming that edge to have a backward scaling factor equal to the forward scaling factor. 4. Replace addition with weighted addition.

[0031] This recipe can be applied fully manually, semi-automatically, or fully automatically. In manual mode, the model designer selects initialization distributions, calculates scaling factors, and inserts these scaling factors into the computation graph according to the recipe above. In semi-automatic mode, parts of this process are automated, but the model designer still needs to (for example) select the implementation of scaled operations and identify cut edges. In fully automatic mode, the model designer can enable unit scaling without providing any additional information, and the system will select appropriate initialization, scaling factors, and identify cut edges automatically.

[0032] After applying the recipe, the method generates a unit-scaled computation graph that can be used to train a deep learning model using gradient-based optimization techniques known in the art.

[0033] At initialization, a deep learning model can choose the initial scale of its parameters. As mentioned in the background above, models typically choose the initialization scale to preserve forward / backward pass scaling. Our technique of unit scaling or target scaling does not require this because scale preservation is built in using a scaling factor; instead, we recommend setting unbiased parameters to have unit scale (standard deviation = 1) at initialization. This specification does not dictate the type of distribution method to use. Bias parameters can be zero-initialized as usual.

[0034] If the inputs to the model are continuous (i.e., not categorical values ​​embedded by the model), they need to be "whitened" to have a zero mean and unit scale. This is a standard procedure: use a sample to estimate the mean and standard deviation, and then use these fixed values ​​to normalize the inputs so that they have the desired statistical value.

[0035] In most cases, forward and backward scaling coefficients can be computed locally for each operation, without the need to propagate information about the input and output distributions through the graph. In this case, we assume that all inputs are independent and normally distributed, with zero mean and unit variance (or a target variance), and derive the output scale through analysis or simulation. The forward scaling coefficient is then set to the inverse of that output scale. The same process is repeated for the backward scale. Examples of scaling coefficients for common operations are shown in Figure 3. Operations include linear operations (e.g., matrix multiplication, sum, weighted addition), activation functions (e.g., ReLU, GeLU, tanh, sigmoid), softmax, softmax cross-entropy, layer normalization, and other operations.

[0036] In some cases, these assumptions may be too strong, and it may be better to assume correlated samples, non-zero mean, etc. This depends on the model used, and after making these assumptions, the same process as above can be applied to derive the scaling factors. An important property of unit (or target) scaling in certain embodiments is that it guarantees correct gradients up to a certain multiplicative factor. To achieve this property, a constrained scaling computation graph is introduced, which constrains the scaling factors with the following rule: for any edge in the forward graph that is not a cut edge, we require that the consuming operation have a backward scaling factor for its input equal to the forward scaling factor. A constrained scaling computation graph that follows this rule will have correct gradients up to a certain multiplicative factor, since it also represents a scaling operation. Such gradients ensure that gradient-based optimization for a unit-scale model is consistent, i.e., there exists an unscaled computation graph that exhibits the same learning dynamics.

[0037] Identifying cut edges in a graph allows manual, semi-automatic, and automatic computation modes. In automatic mode, once the complete forward graph is available, cut edges can be identified by a graph search algorithm. In semi-automatic mode, the model designer defines the model through an API that assumes that parameters are cut edges but activations are not cut edges by default (shared parameters do not imply cut edges, and some activations may be cut edges, so it is optional for the user to override them).

[0038] After computing the scaling factors for the unconstrained cut edges as above, a constraint scaling operation is created by taking each constraint group containing the forward scaling factor and any backward scaling factors corresponding to the unconstrained edges and setting them all equal to the geometric mean of the group. This means that the scales of the output and input gradients may deviate from unit scale, but this tradeoff is necessary to maintain the correct scaled gradients throughout the graph.

[0039] In most cases, in unscaled deep learning models, the scale of tensors at initialization does not play a significant role. A notable exception is when tensors of different scales are added, for example in residual layers, losses, or positional encodings. Naively converting these addition operations to their unit-scale equivalents would result in placing equal weights on the inputs, which could negatively impact performance. Therefore, to solve this, weighted addition is used (see the "weighted_add" operation in Figure 3). This introduces a new hyperparameter to the model, which can be chosen by design principles, empirically via sweep search, or chosen to match a reference model.

[0040] The design principles for residual layers have already been disclosed in the literature. For example, there is the following residual layer based on NF-ResNets (see Brock, A., De, S., Smith, SL and Simonyan, K, "High-Performance Large-Scale Image Recognition Without Regularization," Proceedings of the 38th International Conference on Machine Learning, 2021), which sets activations to x l From x l+1 Convert to. Default:

number

number

number

number

number

number

[0041] Unit scaling is described above as a procedure for building new models, but it can also be applied when there is a need to match the behavior of an existing (baseline) model. There are three main areas where it makes a difference: 1. Nonlinear operations 2. Multi-input operations (excluding multiplication) 3. Optimizer Step Size

[0042] Nonlinear Operations - Deep learning models typically include various nonlinear operations, such as softmax, GELU, and tanh. The behavior of a nonlinear operation can depend on the scale of its input. Because baseline model inputs may not have unit scale, unit-scale model inputs may explore different regions of the nonlinear function, resulting in different behavior. To address this, a scaling factor can be introduced immediately before the activation function (temporarily breaking the unit scale) and a second inverse scaling factor can be removed immediately after (restoring the unit scale). The first scaling factor is chosen to match the empirically or analytically determined input scale of the baseline model, and the second inverse scaling factor is chosen to restore the unit scale when given an input at that scale (also determined empirically or analytically).

[0043] Multi-input operations - operations such as addition are sensitive to the relative scale of the inputs. These scales may vary depending on the input in a baseline model, but should all be approximately 1 in a unit-scale model. To counteract this difference, similar to nonlinear operations, weights (relative scaling factors) must be determined to apply to each input so that the relative contributions of the inputs between the baseline and unit-scale models can be matched while the output maintains unit scale. These weights can be determined empirically or analytically.

[0044] Optimizer step size—Unit scaling guarantees gradients that are scaled versions of the parameter gradients of the unscaled model. Due to this property, the training dynamics (loss and parameter changes with training) may vary between the baseline and unit-scaled models for two reasons. First, optimizers may be sensitive to gradient rescaling, such as SGD (with or without momentum), while Adam is unaffected. Second, models with equivalent training dynamics may differ from the transformed baseline model, especially with reparameterization. To address these two differences, the optimizer step size can be varied per parameter tensor. These step sizes can be calculated analytically by considering the products of all forward scaling coefficients between parameters and loss, and similarly all backward scaling coefficients between loss and parameters.

[0045] The same recipe can be adapted to obtain an arbitrary scale target s. To do so, the source node is modified to produce a tensor of scale s, and an operation is used that preserves scale s given an input of that scale. In the former, parameters are initialized with scale s, and the input is whitened to have scale s. For the latter, linear operations like matmul and weighted_add require no modification, but nonlinear operations require extended analysis or repeated simulations.

[0046] This can be done for new unit-scale models, as described above, or when adapting existing baseline models. This is useful for reducing clipping errors when values ​​are known to shift during training, and quantization errors in numeric formats that do not have a uniform signal-to-noise ratio across the represented range.

[0047] Unit scaling is a procedure applied to a computational graph for training a deep learning model, and its initial implementation cost is low (finding cut edges and calculating scaling factors does not involve significant computations). However, since the goal of using a low-precision format is to save execution time, memory, or both, the execution time and memory efficiency of executing the resulting computational graph are crucial.

[0048] The only modification to the baseline computation graph when using unit scaling is the inclusion of a scaling factor for each forward pass operation and one for each backward pass operation (assuming one backward pass operation per input). For large models with large dense matrices (e.g., Transformer, ResNet), the number of scalar operations (e.g., floating-point operations, FLOPs) for such element-wise scaling operations is negligible. However, the cost of running them as separate kernels on a RAM-attached device, where each kernel involves a round-trip to RAM, can be much larger. Therefore, it is beneficial to consider automatically or manually fusing scaling operations into adjacent kernels to minimize the additional overhead.

[0049] Unit scaling using fused kernels can also reduce the need to write single-precision intermediate values ​​to RAM or the network, since the scaling factor is applied early enough that the values ​​written / communicated are unit-scale, but care must be taken to mitigate the effects of quantization errors.

[0050] As mentioned above, unit (or target) scaling provides the following technical advantages: 1. Aim to achieve a global target scale of tensors (activations, gradients, parameters) during initialization. 2. The ability to choose different scaling factors for the forward pass and the (one or more) backward passes. 3. Local analysis or simulation of the operations to derive scaling factors given an input distribution. 4. Global graph analysis that adds scaling constraints between the forward and backward passes, ensuring that the gradients are correct within a constant factor. 5. Ability to match the training dynamics of existing models using nonlinearity scaling, multi-input weighting, and per-tensor step sizes (but not required). 6. Ability to generate fusion operations with a constant output scaling factor (but not required).

[0051] 4 illustrates an example computer system 100. The computer system 100 is configured to receive as input a computation graph 101. As discussed herein, the computation graph 101 is a graph (e.g., a directed acyclic graph) of operations (i.e., nodes) and tensors (i.e., edges) that describe the structure of a computation. The computation graph 101 may be a forward graph or a backward graph, as discussed herein. The input graph 101 is not scaled.

[0052] The computer system 100 is further configured to execute the recipes / methods described herein above in a fully automated mode. For example, the computer system 100 is configured as follows: 1. Initialize the unbiased parameters with unit variance and whiten the input. 2. Calculate the scaling factors for all (scaled) operations. 3. Identify a non-cut edge and constrain the operation consuming that edge to have a backward scaling factor equal to the forward scaling factor. 4. Replace addition with weighted addition.

[0053] This results in an output graph 102 with unit-scaled tensors. The input graph 101 and / or the output graph 102 may be stored in memory.

[0054] 5 illustrates another example computer system 200 that substantially corresponds to computer system 100 herein, except as described below. Like computer system 100, computer system 200 is configured to receive an input (unscaled) computation graph 201 and output a unit-scaled computation graph 202. However, in contrast to computer system 100, computer system 200 includes a user interface (UI) 210 configured to receive input from a user 215 (i.e., a machine learning model designer).

[0055] The UI 210 is configured to receive user input representing one or more of the following: Selection of one or more initialization distributions for the parameters of the model (i.e., the machine learning model represented by the graph 201). Scaling factors to insert into the computation graph at the relevant positions according to the recipe above. In other words, the user 215 may calculate the scaling factors themselves and use the UI 210 to insert them into the graph at the relevant positions. Explicit specification of cut edge constraints (corresponding to step 3 in the description of computer system 100 above). Selection of one or more weighting hyperparameters for the weighted summation operation (inserted according to step 4 of the description of computer system 100 above). Selection of one or more per-parameter optimizer step size modifiers used to scale the global optimizer step size hyperparameters.

[0056] However, in some examples, the UI 210 is configured to automate at least part of the process. For example, the user 215 may only be required to select an implementation of the scaled operation or to identify a cut edge (or non-cut edge).

[0057] The computer system 200 is then configured to generate a unit-scaled computation graph 202 based on user input received via the UI 210 .

[0058] The UI 210 may broadly consist of any suitable means of interaction with a user 215, including a mouse and keyboard, a display, a touch screen, an audio interface, etc. It also encompasses means for receiving user input via an appropriate network connection (e.g., if the system 200 is a web-based (e.g., cloud-hosted) application accessible via another remote device (e.g., a personal computer) operated by the user).

[0059] 6 illustrates another example computer system 300. The computer system 300 is configured to execute a unit-scaled computation graph 102 / 202 to provide an output 301. In other words, the computer system 300 is configured to execute each operation represented by a node in the graph 102 / 202. The computer system 300 may also receive input data 302.

[0060] In one example, the computation graph 102 / 202 is a graph for training a machine learning model. Thus, the output 301 in such an example is a trained machine learning model resulting from execution of the computation graph. The trained model can take the form of multiple learned parameters, such as a set of learned weights, that are the output of the training process.

[0061] To train a model, the computer system 300 can receive input data 302 in the form of other data and / or parameters in addition to the graph 102 / 202. This can include one or more of a training dataset, hyperparameters for model training, and parameters of pre-trained model components. The hyperparameters can be hyperparameters not already represented in the graph 102 / 202, such as a step size schedule (i.e., learning rate).

[0062] In another example, the computation graph 102 / 202 is a graph for executing a model trained as described above. In other words, the computer system 300 can use the trained model during inference (i.e., in the inference process). In such an example, the graph 102 / 202 is a forward graph, mapping received input data 302 to output 301. The computer system 300 applies forward scaling factors, as discussed herein, and learned parameters, which are the output of the training process, to the graph 102 / 202.

[0063] In such cases, the input data 302 can be broadly considered a query, and the output can be considered a response. The nature of the query and response depends on the task the machine learning model is trained to perform. For example, if the graph 102 / 202 represents a machine learning model trained for image classification, the query can be an input image, and the output can be a classification label. Alternatively, if the graph 102 / 202 represents a machine learning model trained for text classification, the query can be input text, and the output can be a classification label for the text. These are merely examples of suitable trained machine learning models, but the techniques herein are applicable to virtually any input and output modality, and to models other than classification models.

[0064] Each of computer systems 100, 200, 300 includes a suitable processor and memory accessible to the processor. In some examples, the processor includes multiple processing units (e.g., tiles of a tile processor). In some examples, computer systems 100, 200, 300 include multiple processing nodes, each including a processor, which may optionally include multiple processing units. The processing nodes may, in some examples, be arranged in layers.

[0065] Figure 7 illustrates the characteristics of the unit scaling technique discussed herein compared to techniques discussed in the background section of this specification. The "~" indicates that the method ideally requires no tuning, but in practice may introduce hyperparameters that need to be swept. As shown, unit scaling enables fine-grained scaling, with no tuning and low overhead.

[0066] Figure 11 shows common floating-point formats for deep learning that can be used with any of the techniques described herein. E denotes the number of exponent bits, and M denotes the number of mantissa bits for the given format. Max exp. and Min exp. refer to the maximum and minimum values ​​that can be represented by the exponent, excluding special values. E5(a) and E4(a) refer to the FP8 format introduced by Badreddine Noune et al., "An 8-Bit Number Format for Deep Neural Networks," arXiv preprint arXiv:2206.02915, 2022. E5(b) and E4(b) refer to those introduced by Micikevicius et al. (2022).

[0067] 8 shows an example code snippet for implementing certain aspects of the techniques described herein. In particular, function 81 is a scaled_projection function for performing a scaling projection operation. The scaling projection operation implicitly uses β xClass 82 is an unscaled FFN layer of the Transformer model. Class 83 is a unit-scaled FFN layer. Compared to the unscaled layer 82, the scaled FFN layer 83 initializes weights with unit scaling, replaces unscaled operations with scaled operations, replaces residual summation with interpolation according to τ, and shifts the backward pass scale.

[0068] Figure 12 also shows example code for building a unit-scaling model in PyTorch. The scaling function 121 is the basic building block of unit-scaling models. It allows independent control of the scaling coefficients for the forward and backward passes and, as such, must be used with care. This could be used to define a scaled graph with imprecise constraints, leading to gradients that are inconsistent with the model's forward pass. The scaled matrix multiplication function 122 shows how to combine multiple constraints using the geometric mean. The scaled GELU only implements fully constrained scaling for simplicity. When scales are fully constrained, custom gradients via scales are optional. However, it should be noted that they can be useful in some situations to improve the scale of intermediate values. The ScaledLayerNorm 123 class uses the usual assumptions for scaled layers: weights are cut-edge and activations are not. This allows for independent scaling of weight and bias parameters.

[0069] Figure 9 shows the performance of models trained using unit scaling on an example task. In particular, the graph shows unit scaling performance for multiple model architectures and optimizers on the WikiText-103 raw character language modeling task. This task is discussed in Pointer Sentinel Mixture Models by Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher (5th International Conference on Representation Learning [ICLR], 2017). The models are causal language models trained using cross-entropy loss during training and evaluated on bits per character (BPC). Each point on each graph represents a specific combination of sequence layer type, norm placement, and residual scaling, and represents the best final value over a learning rate sweep. All models follow the pattern of the Transformer decoder layer. The sequence layer type is one of the following: attention, RNN, or convolution. The norm placement is one of the following: PreNorm, PostNorm, or NoNorm. Residual scaling is either default, fixed, or running-mean (defined herein). For products of these settings, we compare the performance of normal (baseline) and unit scaling in both FP32 and FP16. For this reason, we also evaluate the normal model with loss scaling in FP16. The hyperparameters used in these experiments are shown in Figure 14.

[0070] The above configuration combination corresponds to 2092 sweeps. First, these results demonstrate the need for scaling when using FP16. This is due to gradient underflow, and loss scaling by a scaling factor of 2048 solves the problem. Second, we demonstrate that unit scaling matches or even slightly improves baseline performance in almost all cases, despite changing the model's training behavior beyond mere numerical computation. Finally, we show that no tuning is required when switching unit scaling to FP16.

[0071] The effect of using different residual scaling schemes was also investigated, and the results are shown in Figure 13 . These results indicate that performance is independent of the choice of method, suggesting that when using unit scaling, either the moving average or fixed method may be an appropriate choice.

[0072] Furthermore, Figure 15 shows further results on a character language modeling task, further demonstrating that the unit-scaled model performs comparable to the regular model and can be trained in FP16 without any modifications or additional hyperparameter selection.

[0073] Figure 10 shows further evaluation of the unit scaling technique discussed here on a masked language modeling task. The standard masked language model pre-training method employed in BERT (Jacob Devlin et al., BERT: Pre-training Deep Bidirectional Transformers for Language Understanding, NAACL-HLT, 2019) is used on English Wikipedia articles. Performance on downstream tasks is then demonstrated in SQuAD v1.1 and SQuAD v2.0 (see: https: / / rajpurkar.github.io / SQuAD-explorer / ). BERT BASE Models and BERT LARGEBoth models are trained for evaluation. F1 stands for F1 score, and EM is a metric that represents the exact match, i.e., the percentage of predictions that exactly match one of the ground truth answers. Figure 16 shows the hyperparameters used in the task in more detail.

[0074] For each model-method-format combination shown in the table, three models were trained, followed by five fine-tuning runs for SQuADv1.1 and SQuADv2.0, for a total of 15 runs per downstream task. The values ​​shown represent the average of the 15 runs, with ± indicating the standard deviation of the mean scores for the three subgroups. The results show that unit scaling yields nearly identical performance in FP16. No performance degradation was observed in FP8 compared to FP16.

[0075] Figures 17 to 20 show the FP16BERT BASE This is a histogram reflecting the absolute tensor values ​​of the model.

[0076] Figure 17 shows the tensor values ​​at the start of the training process for a model trained using loss scaling. Here, we use 2 for stable learning. 15 A loss scale of 1 / 2 was required. Understanding loss scaling in light of this plot shows that we shift the histograms of gradx and gradw by log2(loss scale) to the right. Figure 18 shows the tensor values ​​at the start of the training process for a model trained using the unit scaling technique described here. The first two figures can be understood as the full-model equivalent of the plot in Figure 2B.

[0077] Comparing these two figures clearly shows the effectiveness of unit scaling. While the loss scaling model requires tuning hyperparameters to center the two gradient subplots (grad_xs, grad_ws), the unit scaling model does this naturally. Furthermore, the values ​​of the unit scaling model are generally closer to the center of the range. The loss scaling approach also suffers from very large gradx values ​​in the NSP (next sentence prediction) and MLM (masked language modeling) heads.

[0078] Figures 19-20 show how values ​​shift as a result of training the loss-scale model and the unit-scale model, respectively.

[0079] 21 schematically illustrates a non-limiting example of a computing system 1200 capable of implementing one or more of the methods and processes described above. The computing system 1200 is shown in simplified form. The computing system 1200 may embody any of the computing systems 100, 200, or 300 described above. The computing system 1200 may take the form of one or more personal computers or server computers.

[0080] Computing system 1200 includes a logical processor 1202, a volatile memory 1204, and a non-volatile storage device 1206. Computing system 1200 may optionally include a display subsystem 1208, an input subsystem 1210, a communication subsystem 1212, and / or other components not shown in FIG.

[0081] Logical processor 1202 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical entities. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise obtain a desired result.

[0082] A logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, a logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processors of logical processor 1202 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Individual components of a logical processor may optionally be distributed across two or more separate devices, which may be remotely located and / or configured for cooperative processing. Some of the functionality of a logical processor may be virtualized and performed by remotely accessible networked computing devices configured in a cloud computing configuration. In such cases, it is understood that these virtualized aspects may be executed on different physical logical processors of various different machines.

[0083] Non-volatile storage 1206 includes one or more physical devices configured to hold instructions executable by the logical processor to implement the methods and processes described herein. When such methods and processes are implemented, the state of non-volatile storage 1206 may be transformed, for example, to hold different data.

[0084] Non-volatile storage 1206 can include removable and / or built-in physical devices. Non-volatile storage 1206 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH memory, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), or other mass storage device technologies. Non-volatile storage 1206 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. It is understood that non-volatile storage 1206 is configured to retain instructions even when power to non-volatile storage 1206 is disconnected.

[0085] Volatile memory 1204 may include physical devices including random access memory. Volatile memory 1204 is typically utilized by logical processor 1202 to temporarily store information during the processing of software instructions. It is understood that when power to volatile memory 1204 is removed, volatile memory 1204 typically does not continue to store instructions.

[0086] Aspects of the logical processor 1202, volatile memory 1204, and non-volatile storage 1206 may be integrated together into one or more hardware logic components. Such hardware logic components may include, for example, field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASICs / ASICs), program and application specific standard products (PSSPs / ASSPs), systems on chips (SOCs), and complex programmable logic devices (CPLDs). As noted above, the computer system 1200 may form part of a multi-tile processing device. Many different aspects of a suitable processing device, which may take the form of a chip, are possible. Graphcore has developed an intelligence processing unit (IPU), described, for example, in U.S. Patent Application No. and US Patent Application Publication Nos. 2019 / 0121387, 2019 / 0121388, 2019 / 0121777, and 2020 / 0319861, the contents of which are incorporated herein by reference.

[0087] The terms “module,” “program,” and “engine” may be used to describe aspects of computing system 1200 that are typically implemented in software by a processor to perform a particular function using a portion of volatile memory, where the function includes a transformation that specifically configures the processor to perform the function. Thus, a module, program, or engine may be instantiated via logical processor 1202 using a portion of volatile memory 1204 to execute instructions held by non-volatile storage 1206. It will be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module,” “program,” and “engine” may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.

[0088] If included, display subsystem 1208 may be used to present a visual representation of the data maintained by non-volatile storage 1206. The visual representation may take the form of a graphical user interface (GUI). As the methods and processes described herein modify the data maintained by non-volatile storage and thus transform the state of the non-volatile storage, the state of display subsystem 1208 may likewise be transformed to visually represent the changes in the underlying data. Display subsystem 1208 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with logic processor 1202, volatile memory 1204, and / or non-volatile storage 1206 in a shared enclosure, or such display devices may be peripheral display devices.

[0089] If included, the input subsystem 1210 may consist of or interface with one or more user input devices, such as a keyboard, mouse, touchscreen, or game controller. In some embodiments, the input subsystem may consist of or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and translation and / or processing of input actions may be handled on-board or off-board. Exemplary NUI components may include microphones for speech and / or voice recognition, infrared cameras, color cameras, stereo cameras, and / or depth cameras for machine vision and / or gesture recognition, head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition, electric field sensing components for assessing brain activity, and / or any other suitable sensors.

[0090] If included, communications subsystem 1212 may be configured to communicatively couple the various computing devices described herein with each other and with other devices. Communications subsystem 1212 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, communications subsystem may be configured for communication over a wireless telephone network, or a wired or wireless local or wide area network. In some embodiments, communications subsystem may enable computing system 1200 to send and / or receive messages to and from other devices over a network, such as the Internet.

[0091] Further aspects of the present disclosure and related optional features are described in the following statements, which statements may be combined in any combination, i.e., it is expressly intended that each statement may depend on any other statement.

[0092] According to one aspect of the present disclosure, there is provided a machine learning system that implements a machine learning model, the system comprising at least one tier of processing nodes, each processing node including a processor configured to execute computer-readable instructions to perform at least one operation based on one or more inputs received at the processing node, wherein the at least one operation is scaled by a first scaling factor calculated such that a variance of an output of the at least one operation has a target variance.

[0093] The target variance may be unit variance. The target variance may be a variance that matches the variance of one or more inputs.

[0094] At least one operation is implemented in a forward pass of the machine learning model. The system may be configured to perform a training process for training the machine learning model, the forward pass forming part of the training process. The system may be configured to perform an inference process, the forward pass forming part of the inference process.

[0095] The processing node may be configured to determine a gradient of the loss function in a backward pass of the machine learning model through the layers by performing a gradient calculation in a gradient operation, which may be scaled by a second scaling factor to generate an output having a second target variance.

[0096] One or more of the inputs may include weights, and the gradient calculation may be performed on the weights. One or more of the outputs may include activations, and the gradient calculation may be performed on the activations.

[0097] Any inputs, outputs (e.g., weights and / or activations) discussed herein may be tensors.

[0098] The input may include a set of input activations and a set of weights, and the output may include a set of output activations. The input may include a set of input gradients and a set of weights and / or activations, and the output may include a set of output gradients.

[0099] There may be a weight gradient calculation and an activation gradient calculation. The weight gradient calculation may use a different scaling factor than the activation gradient calculation. One objective of the disclosed technique is to generate a rule set for fixed scaling of operations in the forward and backward passes to maintain the variance of each operation's output to match a target variance, e.g., approximately equal to the variance of that operation's input. This fixed scaling rule set can be applied both at initialization and during training, either alone or in combination with alternative techniques for automatic scaling of signals in the forward and backward passes (e.g., U.S. Patent Application No. 18 / 066,530 (automatic loss scaling) and U.S. Patent Application No. 18 / 066,627 (automatic exponent bias selection)), the contents of which are incorporated by reference.

[0100] In certain embodiments, the system constrains the input and output distributions to have approximately unit variance ("unit scaling"), although this approach can generally be extended to maintain approximately values ​​of input and output variance different from 1 for each operation.

[0101] In one example, a mean of zero and a variance of

number

number

number

number

number

number

number

[0102] In this case, if X and W have unit variance, then to maintain unit variance in the output, Z should be

number

[0103] For the backward pass of the same layer, the loss with respect to activation X

number

number

number

number

number

number

number

number

number

[0104] Similarly, the loss with respect to the weight W

number

number

number

number

number

number

[0105] The scaling factors may be constrained. For example, the scaling factor used in the forward pass operation may be constrained to be equal to the scaling factor used to scale the backward pass gradient computation operation. In certain embodiments, only one of the gradient operations is constrained in its scaling factor, the other being determined by the computation.

[0106] Scaling factors can be calculated for some or all of the operations performed by the neural network. In particular, it may be determined which operations affect the variance of the output relative to the input, and the scaling factors applied to only those operations.

[0107] The scaling factor constraints may only be applied to non-cut edges of the computational graph used to build the machine learning model, and not to cut edges.

[0108] In the fully connected layer example above, the projection operation is typically in the residual block within the shortcut connection. In this situation, the edge connecting the weights W is a cut edge, and the edge connecting the inputs X is not a cut edge. Given this assumption, the techniques herein can constrain the forward pass activation scale α and the backward pass gradient β with respect to activation to be equal, which is equivalent to dividing them by the geometric mean of their unconstrained values.

number

[0109] For backward pass gradients with respect to weights, the techniques herein can instead leave the scale β2 unchanged.

[0110] The system may be configured to execute a computation graph. The computation graph may include a plurality of graph nodes corresponding to computational operations and a plurality of graph edges corresponding to inputs and outputs of the graph nodes. At least one operation corresponds to one of the graph nodes of the plurality of graph nodes of the computation graph.

[0111] The system may be configured to store inputs and / or outputs in 16-bit or less floating-point number representations.

[0112] According to another aspect of the present disclosure, there is provided a computer-implemented method including: receiving a computation graph including a plurality of nodes, each node of the plurality of nodes corresponding to a computational operation for training a machine learning model, and a plurality of edges, each edge connecting a pair of the nodes and corresponding to an output of a first node of the pair of nodes and an input to a second node of the pair of nodes; inserting a first scaling factor into the computation graph associated with at least one of the plurality of nodes, the first scaling factor being calculated such that a variance of an output of the at least one node has a target variance.

[0113] The computation process may be selected from one of a plurality of computation processes, which may be predetermined. The first scaling factor may be selected based on the selected computation process. The computation process and / or scaling factor may be any of those shown in FIG. 3. The first scaling factor may be based on an assumed statistical distribution of the input to the selected computation process.

[0114] The first scaling factor may be a forward scaling parameter that is multiplied with the output of the computational process of at least one node to cause the variance to have a target variance.

[0115] Each node may include a second scaling factor, which is a backward scaling parameter that is multiplied with the result of the gradient operation applied to the node.

[0116] A subset of edges may be cut edges, which are edges that, when cut, cut a pair of nodes connected by the cut edge such that no other path exists between the pair of nodes in the computation graph.

[0117] The method may further include identifying edges other than the cut edges and setting second scaling factors of nodes connected by edges other than the cut edges equal to the first scaling factor.

[0118] The method may include receiving user input via a user interface. The user input may identify a cut edge. The user input may include a first scaling factor or a second scaling factor. The user input may include selecting one or more initialization distributions for parameters of the model, specifying a cut edge constraint, selecting one or more weighting hyperparameters for a weighted summation operation, and / or selecting one or more per-parameter optimizer step size modifiers used to scale a global optimizer step size hyperparameter.

[0119] According to another aspect, a non-transitory computer-readable medium is provided that includes computer-executable instructions that, when executed, implement a neural network, the instructions including first code that implements at least one scaling operation configured to receive a tensor of weights and a tensor of input activations and generate a tensor of output activations having a target variance, where the target variance may be unit variance.

[0120] According to one aspect of the present disclosure, there is provided a machine learning system that implements a machine learning model, the system including at least one layer of processing nodes, each processing node including a processor configured to execute computer-readable instructions, receive a set of input activations and a set of weights, and perform at least one operation to generate a set of output activations, the operation being scaled by a scaling factor calculated such that the variance of the set of output activations generated by the operation has a target variance.

[0121] According to another aspect of the present disclosure, a machine learning system implementing a machine learning model is provided, the system including at least one layer of processing nodes, each processing node including a processor (e.g., one or more processing units) configured to execute computer-readable instructions, receive a set of input activations and a set of weights, and perform at least one operation to generate a set of output activations, the operation being scaled by a scaling factor calculated such that the set of output activations generated by the operation has unit variance.

[0122] According to another aspect of the present disclosure, there is provided a machine learning system that implements a machine learning model, the system including at least one layer of processing nodes, each processing node including a processor (one or more processing units) configured to execute computer-readable instructions, receive a set of input activations and a set of weights, and perform at least one operation to generate a set of output activations, wherein the operation is scaled by a scaling factor calculated such that the variance of the set of output activations generated by the operation matches the variance of the set of input activations.

[0123] Another aspect of the present disclosure provides a method for generating a computer program for implementing a machine learning model (such as a neural network), the computer program including first code embodying at least one scaling operation configured to receive a tensor of weights and a tensor of activations and generate a tensor of output activations having unit variance or a variance that matches the variance of the inputs.

[0124] The computer program may also include second code for implementing one or more scaled gradient calculations for realizing a backward pass of the machine learning model, where each gradient calculation has a scaling factor applied to it to produce an output with unit variance or a variance that matches the variance of the input.

[0125] Another aspect of the present disclosure includes a computer program in the form of transient or non-transient computer-executable instructions, the computer program which, when executed, implements a machine learning model (such as a neural network), the computer program including first code embodying at least one scaling operation configured to receive a vector of weights and a vector of activations and generate a vector of output activations having unit variance or a variance that matches the variance of the inputs.

[0126] The computer program may also include second code for implementing one or more scaled gradient calculations for realizing a backward pass of the machine learning model, where each gradient calculation has a scaling factor applied to it to produce an output with unit variance or a variance that matches the variance of the input.

[0127] The term "unit variance" is used herein in its standard statistical sense to refer to the squared standard deviation of a set of samples, which tends to 1 (unit) as sample size tends to infinity. Variance is determined by the expected value of the squared difference between the sample and the mean of the distribution, which is effectively estimated by calculating the sum of the squared differences between the estimated mean of the sample distribution and the actual sample values, divided by the total number of samples in the distribution.

[0128] When a model is trained, the inputs to the model may be constrained to have unit variance.

[0129] The model has multiple layers, with the output of one layer feeding the subsequent layer.

[0130] While the above aspects and discussion relate to scaling operations that generate output activations, it will be appreciated that these concepts may also be applied to different operations that may be implemented in the context of neural networks, such as deep neural networks. For example, the concepts may be applied to operations performed in attention layers, such as operations involving multiplication of different projections of input activations. They may also be applied to generating weights and / or gradients.

[0131] Accordingly, in another aspect of the present disclosure, there is provided a machine learning system implementing a machine learning model, the system including at least one tier of processing nodes, each processing node including a processor configured to execute computer-readable instructions to perform at least one operation based on one or more inputs received at the processing node, the operation being scaled by a scaling factor calculated such that the variance of an output of the operation has a target variance.

[0132] Any of the methods defined herein may be provided as a computer system or computer readable medium having corresponding features, and vice versa.

Claims

1. at least one tier of processing nodes, each processing node including a processor configured to execute computer-readable instructions to perform at least one operation based on one or more inputs received at the processing node; 10. A machine learning system implementing a machine learning model, wherein the at least one operation is scaled by a first scaling factor calculated such that a variance of an output of the at least one operation has a target variance.

2. The system of claim 1 , wherein the target variance is unit variance.

3. The system of claim 1 , wherein the target variance is a variance that matches the variance of the one or more inputs.

4. The system of claim 1 , wherein the at least one operation is implemented in a forward pass of the machine learning model.

5. The system of claim 4 , wherein the system is configured to perform a training process to train the machine learning model, and the forward pass forms part of the training process.

6. The system of claim 4 , wherein the system is configured to perform an inference process, and the forward pass forms part of the inference process.

7. the processing node is configured to determine a gradient of a loss function in a backward pass of the machine learning model through the layers by performing a gradient calculation in a gradient operation; The system of any preceding claim, wherein the gradient operation is scaled by a second scaling factor to produce an output having a second target variance.

8. The system of claim 7 , wherein the one or more inputs include weights, and the gradient calculation is performed on the weights.

9. The system of claim 7 or 8, wherein the one or more outputs include activations, and the gradient calculation is performed on the activations.

10. The system of any one of claims 1 to 9, wherein the input and the output are tensors.

11. The system of any preceding claim, wherein the input comprises a set of input activations and a set of weights, and the output comprises a set of output activations.

12. The system of any preceding claim, wherein the inputs include a set of input gradients and a set of weights and / or activations, and the outputs include a set of output gradients.

13. The machine learning system is configured to execute a computation graph, the computation graph comprising: a plurality of graph nodes corresponding to computations; a plurality of graph edges corresponding to the inputs and outputs of the graph nodes; The system of any one of claims 1 to 12, wherein the at least one operation corresponds to one graph node of the plurality of graph nodes of the computation graph.

14. The system according to any one of claims 1 to 13, wherein the system is configured to store the input and / or the output in a floating-point number representation represented by 16 bits or less.

15. 1. A computer-implemented method comprising: receiving a computation graph, the computation graph comprising: a plurality of nodes, each node of the plurality of nodes corresponding to a computation for training a machine learning model; a plurality of edges connecting pairs of nodes, each edge corresponding to an output of a first node of the pair of nodes and an input to a second node of the pair of nodes; receiving a computation graph including: inserting a first scaling factor into the computation graph associated with at least one node of the plurality of nodes, the first scaling factor being calculated such that a variance of an output of the at least one node has a target variance.

16. The method of claim 15 , wherein the computational process is selected from one of a plurality of computational processes, and the first scaling factor is selected based on the selected computational process.

17. 17. The method of claim 16, wherein the first scaling factor is selected based on an assumed statistical distribution of inputs to the selected computational process.

18. The first scaling factor is a forward scaling parameter that is multiplied by the output of the computational process of the at least one node so that the variance has the target variance. each node includes a second scaling factor, the second scaling factor being a backward scaling parameter that is multiplied with a result of the gradient operation applied to the node; the subset of edges are cut edges, which, when cut, cut the pair of nodes connected by the cut edge such that no other path exists between the pair of nodes in the computation graph; The method comprises: identifying an edge other than the cut edge; setting the second scaling factor of nodes connected by edges other than the cut edge equal to the first scaling factor; The method of any one of claims 15 to 17, further comprising:

19. The method of claim 18, comprising receiving user input via a user interface specifying the cut edge.

20. The method of any one of claims 15 to 19, comprising receiving the first scaling factor via a user interface.

21. A non-transitory computer-readable medium containing computer-executable instructions that, when executed, implement a neural network, A non-transitory computer-readable medium, the instructions including first code embodying at least one scaling operation configured to receive a tensor of weights and a tensor of input activations and generate a tensor of output activations having a target variance.

22. 22. The non-transitory computer-readable medium of claim 21, wherein the target variance is unit variance.

Images