System for training artificial neural networks

Abstract

An embodiment of the present disclosure includes a system for optimizing an artificial neural network by configuring a model based on a plurality of training parameters to perform a training process, monitoring a plurality of statistics resulting from the execution of the training process, and adjusting one or more of the training parameters based on one or more of the statistics to keep at least one of the statistics within a predetermined range. In some embodiments, an artificial intelligence (AI) processor can perform a training process for the model, the training process having an associated set of training parameters. The execution of the training process can result in a plurality of statistics. A control processor coupled to the AI ​​processor can receive the statistics and adjust one or more of the training parameters accordingly during the execution of the training process to keep at least one of the statistics within a predetermined range.

Description

[Technical Field]

[0001] background

[0001] The present disclosure relates to computing systems. More particularly, the present disclosure relates to techniques for training artificial neural networks. [Background technology]

[0002]

[0002] Artificial intelligence (AI) systems have enabled significant advances in various fields, such as natural language processing and computer vision. AI systems typically include AI models (e.g., neural network models) that are composed of multiple layers. Each layer typically includes nodes (also known as neurons) that are connected to nodes in other layers. The connections between nodes are associated with trainable weights to increase or decrease the strength of the connections. Operationally, a data set is applied to the input layer of the model, and an output is generated at the output layer. The output may correspond to the classification, recognition, or prediction of a particular feature of the input data set. To train a neural network, the output is compared to known outputs for the input data set, the error is backpropagated through the model, and the model's parameters are adjusted.

[0003] One problem with neural network models is that larger models typically require more computational resources and / or time to process, primarily due to the number of parameters associated with such models that require computation.

[0004] BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Various embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings. [Brief explanation of the drawings]

[0005] [Figure 1] 1 illustrates a system for training an artificial neural network, according to one embodiment. [Figure 2]

[0006] 1 illustrates a method for training an artificial neural network, according to one embodiment. [Figure 3A]

[0007] 1 illustrates training parameters and monitored values ​​in an artificial neural network, according to one embodiment. [Figure 3B]

[0008] 1 illustrates a mapping system for training an artificial neural network, according to one embodiment. [Figure 4]

[0009] 1 illustrates a method for training an artificial neural network according to another embodiment. [Figure 5A]

[0010] 1 illustrates a system for generating a mapping by training a smaller network, according to one embodiment. [Figure 5B]

[0011] 1 illustrates a system for loading mappings for training a larger network, according to one embodiment. [Figure 6]

[0012] 1 illustrates an example of a neural network that may yield statistics, according to one embodiment. [Figure 7]

[0013] 1 shows a graph illustrating an example of monitored statistics, according to one embodiment. [Figure 8]

[0014] 1 illustrates an example of adjusting the precision associated with various portions of a neural network, according to one embodiment. [Figure 9]

[0015] 1 illustrates an example of adjusting the sparsity associated with various portions of a neural network, according to one embodiment. [Figure 10]

[0016] 1 illustrates an example of adjusting the size of models associated with different parts of a neural network, according to one embodiment. [Figure 11]

[0017] 1 illustrates a method for using an AI processor to scale an artificial neural network, according to one embodiment. [Figure 12]

[0018] FIG. 1 illustrates a simplified block diagram of an example computer system, according to one embodiment. [Figure 13]

[0019] 1 illustrates an artificial neural network processing system, according to one embodiment. [Figure 14]

[0020] 1 illustrates an example of a graph showing optimization of validation loss versus computational resources and time, according to one embodiment. [Figure 15A]

[0021] 10 shows a graph illustrating gradient noise introduced by a high accuracy model, according to one embodiment. [Figure 15B]

[0022] 10 shows a graph illustrating gradient noise introduced by a non-optimized low precision model, according to one embodiment. [Figure 15C]

[0023] 10 shows a graph illustrating gradient noise being controlled in an optimized low-precision model, according to one embodiment. [Figure 15D]

[0024] 1 shows graphs illustrating the convergence of a high precision model, the divergence of a non-optimized low precision model, and the convergence of an optimized low precision model, according to one embodiment. [Figure 16A]

[0025] 10 shows a graph illustrating the convergence of a high precision model and the divergence of a non-optimized low precision model, according to one embodiment. [Figure 16B]

[0026] 10 shows a graph comparing layer activations provided by a high precision model with layer activations provided by a non-optimized low precision model, according to one embodiment. [Figure 16C]

[0027] 10 shows a graph comparing the bed slopes produced by a high accuracy model with the bed slopes produced by a non-optimized low accuracy model, according to one embodiment. [Figure 17A]

[0028] 10 shows a graph illustrating validation loss for models with different accuracies, according to one embodiment. [Figure 17B]

[0029] 10 shows a graph comparing a first example of the percentage of zero values ​​in layer activations produced by models with different accuracy, according to one embodiment. [Figure 17C]

[0030] 10 shows a graph comparing a second example of the percentage of zero values ​​in layer activations produced by models with different accuracy, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0006] Detailed Description

[0031] In the following description, for purposes of explanation, numerous examples and specific details are set forth to enable a thorough understanding of the present disclosure. Such examples and details should not be construed as unduly limiting the elements of the claims or the entire claimed subject matter. Based on the language of the various claims, it will be apparent to those skilled in the art that the claimed subject matter can include some or all of the features in these examples, alone or in combination, and can further include modifications and equivalents of the features and techniques described herein.

[0007]

[0032] In deep learning, artificial intelligence (AI) models (e.g., neural network models) typically improve their prediction accuracy as their size (e.g., number of layers, nodes, connections, and / or the like) increases. This accuracy is often measured during training as a desirable reduction in validation loss (e.g., more accurate predictions).

[0008]

[0033] However, increasing the size of a model typically requires increased computational resources and / or time to process the model due to the increasing number of parameters associated with the increased model size, which requires additional computation.

[0009]

[0034] For example, for each node in a neural network (NN) model, y=f(x0w0+x1w1+....+x n w n), where y represents the node's output value, x represents the input value from connecting nodes 0 through n, and w represents the trainable weights (e.g., parameters) associated with the connections from the node. During training, the model's output (e.g., in the last layer) can be compared to known outputs for the input dataset. A similar backward pass computation (e.g., backpropagation) can then be performed to determine gradient and weight updates. For example, in a process known as stochastic gradient descent (SGD), backpropagation can be performed multiple times (e.g., iteratively) on subsets of the training dataset. The computations in the forward and backward passes are typically performed via matrix multiplication (also known as Mat-Mul) operations, which are performed many times per layer of the model. As a result, the number of computations required to train a model can quickly increase as the size of the model increases.

[0010]

[0035] One technique for reducing the computational resources and / or time for processing a model is to reduce the precision (also known as bitwidth) of the calculations associated with the model. For example, the computational resources and / or time for processing a model requiring high-precision calculations, such as IEEE 754 single-precision 32-bit floating-point format (also known as FP32), can be reduced when compared to the computational resources and / or time for processing a model involving low-precision calculations, such as 8-bit integers (also known as INT8).

[0011]

[0036] However, once accuracy falls below a certain threshold, models often fail to improve their prediction accuracy. Furthermore, models may even worsen their prediction accuracy with continued training (e.g., divergence).

[0012]

[0037] Features and advantages of the present disclosure include improving the training of neural network models by automatically adjusting one or more training parameters associated with the model based on one or more statistics associated with the model to keep certain statistics within predetermined ranges. In some embodiments, adjusting training parameters based on statistics associated with the model may be referred to as “introspective” training. The techniques disclosed herein may support improved training performance that scales linearly with model size, with reduced scaling of computational resources, for example. Some embodiments of the present disclosure therefore provide an automatically scaling introspective form of training for neural network models. Advantageously, such training may enable lower validation loss (e.g., improved prediction accuracy) toward convergence, while resulting in reduced computational resources and / or time for processing (e.g., reduced computational cycles) for very large models. Certain embodiments described below may further include automated techniques for adjusting parameters (e.g., based on statistics).

[0013]

[0038] FIG. 1 illustrates a system 100 for optimizing artificial neural networks, according to one embodiment. In this example, one or more control processors 102 may communicate with one or more AI processors 104. The control processor 102 may include, for example, a conventional CPU, FPGA, system-on-chip (SoC), application-specific integrated circuit (ASIC), or embedded ARM controller, or other processor capable of executing software and communicating with the AI ​​processor 104 based on instructions in the software. The AI ​​processor 104 may include a graphics processor (GPU), an AI accelerator, or other digital processor optimized for AI operations (e.g., matrix multiplications versus Von Neumann Architecture processors, such as x86 processors). Examples of AI processors may include, for example, a GPU (e.g., an NVidia Volta® processor with 800 cores and 64 multi-accumulators) or a Tensor Processor Unit (TPU) (e.g., four cores with 16k parallel operations).

[0014]

[0039] In this example, the control processor 102 may be coupled to a memory 106 (e.g., a non-transitory computer-readable storage medium) that stores program code executable by the control processor 102. The control processor 102 receives (e.g., loads) a neural network model 110 (hereinafter "model") and a number of training parameters 112 for configuring the model 110. The model 110 may include, for example, a graph defining multiple layers of a neural network, with nodes in a layer connected to nodes in other layers and connections between nodes associated with trainable weights. The training parameters 112 (also known as tuning parameters or model parameters) may include one or more values ​​that can be adjusted to affect the configuration and / or performance of the model 110. As will be apparent to those skilled in the art in light of this disclosure, training parameters that may be used in various embodiments include, for example, model size, batch size, learning rate, precision (e.g., the number of bits in a binary representation of a data value), and sparsity (e.g., the number of zeros in a data value), regularization (e.g., weight decay or L2 regularization), entropy, and / or training steps, as well as other parameters that can be characterized and adjusted. As known to those skilled in the art, in some embodiments, training parameters may include one or more hyperparameters (e.g., parameters used to control the learning of a neural network).

[0015]

[0040] The control processor 102 may also execute a neural network compiler 114. The neural network compiler 114 may include a program that, when executed, receives the model 110 and training parameters 112 and configures resources 105 on one or more AI processors to implement and execute the model 110 in hardware. For example, the neural network compiler 114 may receive the model 110 and configure the model 110 based on the training parameters 112 to execute a training process executed on the AI ​​processor 104. To execute the training process, the neural network compiler 114 may configure the AI ​​processor 104 to implement calculations such as input activations, weights, backpropagation, etc. In response, the AI ​​processor 104 may receive and process training data 116 along with the model 110 (e.g., the training process) using resources 105 determined by the neural network compiler 114. For example, the resources 105 may include registers, multipliers, adders, buffers, and other digital blocks used to perform operations to implement the model 110. Thus, the AI ​​processor 104 may perform numerous matrix multiplication calculations in a forward pass, compare the output to known outputs for a subset of the training data 116, and perform further matrix multiplication calculations in a backward pass to determine, for example, gradient or weight updates. This process may continue through multiple iterations as the training data is processed. In some embodiments, the AI ​​processor 104 may determine weight updates according to a backpropagation algorithm, such as stochastic gradient descent (SGD), adaptive moment estimation (ADAM), and / or the like, which may be configured by the neural network compiler 114.

[0016]

[0041] During execution of the model 110, the AI ​​processor 104 may generate multiple values ​​for, e.g., activations, weights, gradients, and weight updates at each layer of the model at each iteration. Features and advantages of the present disclosure include monitoring statistics of such values ​​at certain locations within the neural network and adjusting training parameters 112 (e.g., layer by layer) based on such statistics to improve training. As described in more detail below, the monitored statistics may include, e.g., one or more of backpropagation gradients, gradient noise, measurements of node activations, weights, weight updates, and / or the like. The statistical measurements may include, e.g., measurements of the mean, measurements of the standard deviation, measurements of the percentage of zero values, and / or the like.

[0017]

[0042] In some embodiments, the control processor 102 may execute a mapping system 132. The mapping system 132 may, for example, process statistics to automatically adjust one or more training parameters 112 associated with a portion of the model, thereby keeping one or more statistics within predetermined ranges (e.g., at a particular location within the model).

[0018]

[0043] The adjusted training parameters 112 can be provided to the neural network compiler 114 to update the implementation of the model 110 on the AI ​​processor 104 for subsequent execution by the AI ​​processor. This process can be repeated in iterations for subsets of the training data 116 until the model 110 converges after a predetermined number of training steps. Ultimately, a trained model 120 can be obtained, for example, for deployment in a given application. Advantageously, by adjusting the training parameters 112 to keep statistics within predetermined ranges, the resulting trained models can be optimized to improve their predictive accuracy, for example, while requiring less computational resources and / or time to process (e.g., reducing computational cycles).

[0019]

[0044] In some embodiments, different training parameters can be used for different batches of training data used to train the same model. This can be referred to as hybrid mode. For example, for different batches running on different processors (or processors operating on the same training batch), different precision can be used for different parts of the model being trained. Hybrid mode can be an extension with the ability to distribute training across multiple different training nodes, for example, running each node at its own precision. For example, when reducing precision, gradient noise can increase, which can lead to training divergence and suboptimal performance. In some embodiments, adding several high-precision guide nodes to a distributed training run is an effective way to reduce overall gradient noise and improve model loss. In a data-parallel setting, this can be thought of as sampling 1 through M of a total of N batches with high precision, where, for example, N< <Mである。

[0020]

[0045] As indicated above, in some embodiments, the model 110 may be partitioned to run on multiple AI processors 104. For example, a first portion of the model 110 may run on a first AI processor, and a second portion of the model 110 may run on a second AI processor. In this example, measurements of multiple values ​​may be received from multiple AI processors for analysis by one or more control processors 102. In various embodiments, this configuration may allow for efficient distribution of the training process for faster deployment while optimizing the model 110. In various embodiments, value-based statistics may be generated by the AI ​​processor or the control processor, for example.

[0021]

[0046] FIG. 2 illustrates a method for training an artificial neural network according to one embodiment. In this example, at 202, one or more control processors may receive a neural network model for a given application. The control processor may be such as control processor 102. At 204, the control processor may configure the model based on the training parameters to execute a training process on one or more AI processors. The AI ​​processor may be such as AI processor 104. At 206, the control processor may monitor a plurality of statistics resulting from execution of the training process by the AI ​​processor. At 208, the control processor may determine whether one or more statistics are within a predetermined range. If one or more statistics are within the predetermined range (yes), at 206, the control processor may continue to monitor the plurality of statistics in a loop until the model converges after a predetermined number of training steps. However, if one or more statistics are not within the predetermined range (no), at 210, the control processor may adjust the training parameters based on the statistics to keep the one or more statistics within the predetermined range. Returning to 204, the control processor may then configure the model based on the adjusted training parameters to execute a training process on the AI ​​processor.

[0022]

[0047] FIG. 3A illustrates training parameters and monitored values ​​within an artificial neural network, according to one embodiment. In this example, a model 300 is loaded onto one or more AI processors 304 to perform a training process. The AI ​​processors 304 are further configured with the following training parameters: model size 310, batch size 311, learning rate 312, precision 313, and sparsity 314. As will be described in more detail below, different configurations of the various training parameters can alter the behavior of the neural network model during training. For example, a training process can be configured to run the model 300 with different model sizes, batch sizes, learning rates, precision, and sparsity. As will be further described below, altering the training parameters results in different values ​​occurring throughout the network being trained. The values ​​of gradient noise 301, activations 302, gradients 303, weights 304, and weight updates 305 can be monitored at specific locations within the network and at specific points during the training process and used to modify the training parameters. A feature and advantage of the present disclosure is that certain training parameters (e.g., learning rate, accuracy, sparsity) at particular locations within the model (e.g., in one or more particular layers) can be configured to have particular values ​​that result in improved training performance. Similarly, training parameters can be changed over time, for example, to optimize the training process, reduce computation, and / or improve accuracy.

[0023]

[0048] 3B illustrates a mapping system for training an artificial neural network, according to one embodiment. In this example, mapping system 332 may include a statistics monitor 340, mapping rules 342, and training parameter adjustment output 344. Mapping system 332 may be an example of mapping system 132, for example.

[0024]

[0049] In some embodiments, a statistics monitor 340 can receive values ​​associated with a given model running on one or more AI processors, perform statistical processing (e.g., determining the mean, standard deviation, or percentage of zeros), and send the statistics to mapping rules 342. Values ​​generated during model execution can be received from AI processors running various portions of the model at various times during the AI ​​processor's execution of the training process. In this example, statistics can include, for example, measurements of gradient noise 301, activations 302, backpropagation gradients 303, weights 304, and weight updates 305 taken at a particular layer in the model being trained. As noted above, statistical measurements can include, for example, one or more of the mean, standard deviation, and / or percentage of zero values ​​(or equivalently, non-zero values). Mapping rules 342 specify a relationship between one or more monitored statistics and one or more training parameters. Mapping rules can be generated, for example, by characterizing the training of a model using a smaller model size to establish relationships between statistics and training parameters and define mapping rules for controlling training parameters based on the statistics to optimize training. These statistics can then be used to dynamically adjust the model size 310, batch size 311, learning rate (e.g., at a particular position or time), precision 313 (e.g., at a particular node or layer), and sparsity 314, e.g., to improve training accuracy and / or increase the computational efficiency of the training process.

[0025]

[0050] Embodiments of the present disclosure can monitor gradient noise 301 at specific locations within a neural network model and adjust training parameters to control the gradient noise within a predetermined range. As known to those skilled in the art, gradient noise refers to noise in the gradient of backpropagation. For example, gradient noise can be expressed as the signal-to-noise ratio of the gradient. In some cases, gradient noise may be related to quantization effects (e.g., representing the gradient as a binary value). Thus, gradient noise can be a function of the quantization noise-to-signal ratio (QNSR), e.g., the signal is the ideal gradient value. Improved training can result when there is a "huge" amount of gradient noise at certain nodes or layers within the network. Thus, embodiments of the present disclosure can monitor gradient noise at specific locations within the network and / or at specific times during training and adjust certain training parameters to keep the gradient noise within a predetermined range.

[0026]

[0051] For example, gradient noise is a function of batch size, with small batches resulting in more gradient noise and large batch sizes resulting in less gradient noise (e.g., the gradients of the batch are closer to the ideal gradient). As another example, quantization noise (e.g., QNSR) decreases with increasing precision (e.g., number of bits) and increases with decreasing precision. As quantization noise increases, so does gradient noise. Thus, for example, training parameters that affect gradient noise in a network can be adjusted during training based on the gradient noise monitored at specific parts of the model (e.g., nodes or layers) to keep the gradient noise at those locations and / or times within an optimal range.

[0027]

[0052] Similarly, to control the model size 310, batch size 311, learning rate 312, accuracy 313, or sparsity 314, statistics regarding gradient noise 301, activation 302, gradient 303, weight 304, and weight update 305 can be mapped individually or in various combinations. Advantageously, the learning rate 312, accuracy 313, and / or sparsity 314 can target specific portions of the network model such that, for example, different portions of the trained model are configured differently in order to optimize training. Thus, in some examples of embodiments, the monitored values can be kept within an optimal range (e.g., a > x, a < x < b, x < a) to facilitate efficient training as indicated by the controlled statistics C1...CN.

[0028]

[0053] FIG. 4 shows a method of training an artificial neural network according to one embodiment. In this example, a small neural network model (e.g., fewer nodes, layers, or weights) can be trained to train a large neural network model in order to efficiently characterize the relationship between the monitored statistics and the training parameters.

[0029]

[0054] At 402, one or more control processors can receive a first neural network model having a first model size (e.g., a small size) for a given application. At 404, the control processor can configure the first model based on training parameters to execute a first training process on one or more AI processors. At 406, the control processor can monitor a plurality of statistics at various positions of the first model resulting from the execution of the first training process by the AI processors. In some embodiments, the control processor can monitor how the statistics change based on a change in the value (or setting) of the training parameters. At 408, the control processor can generate one or more mappings characterizing the relationship between the statistics and the training parameters (e.g., how the statistics change based on a change in the training parameters).

[0030]

[0055] In various embodiments, the mapping may characterize the relationship between statistics and training parameters in various ways. For example, the mapping may characterize the relationship between adjustments to model size, batch size, learning rate, accuracy, sparsity, weight decay, training steps, etc. (e.g., training parameters) and measurements of backpropagation gradients, gradient noise, node activations, weights, weight updates, etc. (e.g., statistics). In certain embodiments, the mapping may characterize the relationship between model size and gradient noise, between batch size and gradient noise, between batch size and backpropagation gradients, between learning rate and backpropagation gradients, between learning rate and gradient noise, between accuracy and backpropagation gradients, between accuracy and gradient noise, between weight decay and activations, between weight decay and gradient noise, between learning rate and activations, sparsity and gradient noise, etc. In some embodiments, the characterization may be empirically determined during training of a smaller model (e.g., a first model) to efficiently apply to a larger model. In some embodiments, the mapping may be based on empirical measurements of statistics. Examples of characterizations and mappings between statistics and training parameters are provided below.

[0031]

[0056] Returning to the example of Figure 4, at 410, the control processor may receive a second neural network model having a size (e.g., a large size) of the second model. At 412, the control processor may load the mapping generated at 408 and configure the second model to perform a second training process on the AI ​​processor using training parameters adjusted according to the mapping. Advantageously, the AI ​​processor may be configured to perform a second training process for the second model based on the characterizations from the first training process for the first model.

[0032]

[0057] The second model may be ready for training by the AI ​​processor. Thus, at 414, the control processor may monitor multiple statistics at certain positions within the second model resulting from execution of the second training process by the AI ​​processor. At 416, the control processor may determine whether one or more statistics are within a predetermined range. If one or more statistics are within the predetermined range (yes), at 414, the control processor may continue to monitor the multiple statistics during the training process. However, if one or more statistics are not within the predetermined range (no), at 418, the control processor may adjust training parameters based on the statistics to keep the one or more statistics within the predetermined range. Then, at 420, the control processor may configure the model based on the adjusted training parameters to execute the second training process and return to 414. In various embodiments, the first model may be significantly smaller (e.g., by several orders of magnitude) than the second model so that characterization is computationally faster and more feasible on smaller models and computationally impractical on larger models. Advantageously, by characterizing the relationship between the monitored statistics and the training parameters of the first model for subsequent application to the second model, reduced computational resources and / or time (e.g., reduced computational cycles) for processing in connection with training the second model may be realized.

[0033]

[0058] In one example, FIG. 5A illustrates a system configured to generate a mapping by training a first model 510a, and FIG. 5B illustrates a system configured to load the mapping to train a second model 510b, according to one embodiment. In this example, as shown in FIG. 5A, a control processor 502 can receive a first model 510a having a size (e.g., a small size) for a given application. The control processor 502 can configure the first model 510a based on training parameters 512 to run the first training data 516a on one or more AI processors 504. The control processor 502 can monitor statistics at each layer of the first model 510a resulting from execution of the training process using the first training data 516a. In this example, the control processor 502 can implement a mapping generator 532a to generate one or more mappings characterizing the relationship between statistics and training parameters.

[0034]

[0059] In some embodiments, a mapping system can maximize model performance for a given budget. For example, a mapping generator 532a can be used to create a mapping system 532b by fitting a custom model that observes how the behavior of various network hyperparameters, such as model accuracy, size, and gradient noise, affects training statistics during the training of a single or series of small networks. The mapping generated from previous small runs can be combined with training statistics and model performance data. This information is used to train a model (e.g., a neural network model) that predicts model performance given a set of input hyperparameters, including model size, model accuracy, and the like. This mapping system can be used to generate optimal hyperparameters for a larger model. For example, the mapping system can predict model performance based on size and computational budget and indicate to the user the confidence level of the prediction. As the mapping system is automatically trained from previous runs, no further user input may be required, eliminating the need to manually tune various hyperparameters of a deep learning model. Thus, in certain example embodiments, power laws and typical model statistics can be used as automated probabilistic tools that can predict, for example, the behavior of a model (e.g., in terms of accuracy) in a zero-shot setting for each configuration (e.g., various accuracy or sparsity).

[0035]

[0060] As shown in FIG. 5B , control processor 502 can receive second model 510b having a second model size (e.g., the size of the large model). Control processor 502 can implement mapping system 532b to load the mapping generated by mapping generator 532a. Mapping system 532b can then configure second model 510b for execution of a second training process on AI processor 504 using training parameters 512 and training data 516b. Advantageously, to improve training performance, the same parameters used to determine the mapping using the smaller model can now be adjusted based on monitored statistics within the large model. In particular, control processor 502 can monitor statistics at various locations within second model 510b resulting from execution of the second training process (e.g., the same statistics at the same layers as in the small model). If one or more statistics are not within a predetermined range, control processor 502 can adjust one or more of second training parameters 512 based on measurements of the monitored statistics to keep one or more statistics within the predetermined range. By characterizing the relationship between the monitored statistics and the training parameters for the first (smaller) model 510a for later application to the second (larger) model 510b, reduced computational resources and / or time (e.g., reduced computational cycles) for processing when training the second model 510b can be realized.

[0036]

[0061] It should be understood that in various embodiments, the mapping generator step of FIG. 5A and the statistical monitoring and training parameter adjustment step of FIG. 5B may be performed on the same or different control processors 502.

[0037]

[0062] FIG. 6 illustrates an example of a neural network 600 that can produce statistics, according to one embodiment. In this example, neural network 600 can represent a neural network model, such as model 110, that can be configured with training parameters to execute a training process on one or more AI processors. Neural network 600 can include, for example, a graph defining “n” layers (labeled L1 through Ln, where n is an integer) of a neural network, where nodes in a layer are connected to nodes in other layers and trainable weights (e.g., parameters) are associated with the connections between the nodes. The first layer (L1) can be an input layer to which training data can be applied. The last layer (Ln) can be an output layer that can generate output. Layers between L1 and Ln (e.g., L2, L3, etc.) can be hidden layers between the first layer (L1) and the last layer (Ln). Layers closer to the first layer (L1), such as the second layer (L2) (e.g., layers closer to or toward the input relative to the midpoint of the neural network), can be referred to as lower or preceding layers. Layers closer to the output layer (Ln), such as layer n-1 (not shown) before the last layer (e.g., layers closer to or toward the output relative to the midpoint of the neural network), may be referred to as upper or backward layers.

[0038]

[0063] In one example, for each node in the model, the AI ​​processor computes y=f(x0w0+x1w1+....+x n w n), where y represents the node's output value, x represents the input value from connecting nodes 0 through n, and w represents the trainable weights (e.g., parameters) associated with the connections from the node. During training, the model's output (e.g., at the output layer) can be compared to known outputs for the corresponding input data set. The AI ​​processor can then perform a backward pass calculation (e.g., backpropagation) (labeled backward pass) to determine gradient and weight updates. For example, the above calculations can be performed using matrix multiplication (also known as Mat-Mul) operations. Activations and weights can be processed using multipliers and adders (e.g., multiply-adders), which can be configured to have a particular precision (e.g., a particular number of bits and a particular binary representation).

[0039]

[0064] During execution of the model, the AI ​​processor can provide measurable values ​​at specific points within the neural network at each iteration and generate statistics from those values. As described above, measurements can include activation values, weight values, gradient values, gradient noise values, and weight update values. In this example, for a particular layer of the model (e.g., layer by layer), statistics of the measurements (described above), including values ​​at hidden layers, resulting from either or both forward and backward passes, can be determined. For example, during forward and backward passes, the last layer (e.g., output layer or Ln) can provide Ln statistics for the first layer (e.g., input layer or L1), which can provide Ln statistics, and so on. As described above, in one example embodiment, a control processor can receive measurements for each layer of the model and generate statistics. Advantageously, monitoring statistics at individual layers can provide the ability to control statistics at specific target layers by adjusting one or more training parameters.

[0040]

[0065] Although this example shows a feedforward neural network, it should be understood that the techniques described herein may be applicable to improve the training of other neural network topologies.

[0041]

[0066] FIG. 7 shows a graph 700 illustrating example monitored statistics for a model according to one embodiment. This example illustrates how controlling the range of certain statistics in particular portions of a neural network model can improve training performance. In this example, the average gradient noise is monitored in each layer of an example 12-layer Transformer model (L0-L11) in a natural language processing (NLP) application. Here, training is improved by adjusting training parameters so that later layers (e.g., layers closer to the output) advantageously have higher gradient noise than earlier layers (e.g., layers closer to the input). As shown in FIG. 7, the last layer (L11) of the model may have the highest gradient noise at 702, the first layer (L0) of the model may have the lowest gradient noise at 704, and layers in between (L1-L10) may have decreasing gradient noise levels according to each layer's decreasing layer position in the model. Controlling later layers to have higher gradient noise than earlier layers may advantageously improve performance and convergence, with a corresponding reduction in computational cycles, for example, in NLP. However, for other models for applications in other (e.g., non-NLP) domains, the optimal gradient noise and other statistical distributions across the model may be different, and may be obtained, for example, by the characterization techniques described above.

[0042]

[0067] FIG. 8 illustrates an example of adjusting precision associated with various portions of a neural network 800, according to one embodiment. Generally, higher precision (e.g., floating-point 16 or 32) increases the accuracy of the model, but also increases computational resources. Features and advantages of the present disclosure include adjusting precision (e.g., the number of bits representing a value) at various portions or locations of the model to improve training. Thus, a model undergoing a training process can be configured with high precision at some locations (e.g., specific layers or nodes) and low precision (e.g., 8-bit integers) at other locations where precision may not be necessary. For example, as previously shown, precision can affect gradient noise. More specifically, precision affects quantization noise (e.g., a smaller number of bits representing a value increases quantization noise, which in turn increases gradient noise). It may be desirable to keep gradient noise within a certain range across the model being trained or in specific portions of the model (e.g., at specific layers or specific nodes). As shown in the example of FIG. 7, it may be desirable to adapt the gradient noise across various layers. Thus, to improve training, gradient noise can be monitored, and the precision used in each layer can be used to keep the gradient noise within a predetermined range. 8, for example, layers closer to the input (e.g., L2 and L3) can be configured with higher accuracy than layers closer to the output (e.g., Ln-1 and Ln-2). More specifically, features and advantages of the present disclosure may model gradient noise as a function of quantization noise-to-signal ratio (QNSR) and other hyperparameters to obtain robust and accurate training that reduces the computational resources required for training.

[0043]

[0068] FIG. 9 illustrates an example of adjusting sparsity associated with various portions of a neural network 900, according to one embodiment. In this example, a control processor, such as control processor 102, can adjust the sparsity associated with one or more target layers and / or nodes of the model. The control processor can adjust the sparsity to keep one or more statistics associated with the target layers and / or nodes within a predetermined range. For example, the model may be initially configured with a low sparsity, resulting in fewer calculations being skipped and fewer values ​​being forced to zero. Based on one or more statistics, such as low gradient noise in the second layer (L2), the control processor can adjust the sparsity associated with the second layer (L2) to provide a high sparsity, resulting in more calculations being skipped and more values ​​being forced to zero, while the other layers remain at the low sparsity initially configured. This adjustment of the second layer (L2) may enable one or more statistics in the second layer (L2) to be kept within a predetermined range. Continuing with this example, the control processor may additionally or alternatively adjust the sparsity associated with a given node in a target layer, such as the last node in the third layer (L3). This adjustment in the third layer (L3) may enable one or more other statistics associated with the third layer (L3) to remain within predetermined ranges. It should be understood that many different configurations are possible for adjusting sparsity differently in various portions of the model.

[0044]

[0069] FIG. 10 illustrates an example of adjusting the size of a model associated with various portions of neural network 1000, according to one embodiment. In this example, the control processor may adjust the number of nodes associated with a target layer and / or the number of layers associated with the model. The control processor may adjust the size of the model to keep the target layer and / or one or more statistics that may be associated with the nodes in the target layer within a predetermined range. For example, the model may initially be configured with n layers (L1 to Ln). Based on one or more statistics, the control processor may adjust the number of layers by adding a new layer adjacent to the third layer (L3) (e.g., an additional fourth layer (L4) before or after the layer of the measured statistics), thereby adjusting the model to n+1 layers (L1 to Ln+1). This adjustment of the model size (e.g., an additional fourth layer (L4)) may allow one or more statistics to remain within a predetermined range. Continuing with this example, the control processor may additionally or alternatively adjust the number of nodes associated with the target layer, such as by adding to the number of nodes in the second layer (L2). This adjustment within the second layer (L2) may allow one or more other statistics associated with the second layer (L2) to remain within predetermined ranges. It should be understood that there are many different configurations possible for adjusting the size of the model differently in various parts of the model.

[0045]

[0070] FIG. 11 illustrates a method for using an AI processor to scale an artificial neural network, according to one embodiment. In this example, at 1102, one or more AI processors may be configured to perform a training process associated with a set of training parameters on a neural network model. The AI ​​processor may be such as AI processor 104. At 1104, the AI ​​processor may perform the training process to generate values ​​having multiple statistics in various portions of the model. At 1106, the system may determine whether one or more statistics fall within one or more predetermined ranges. If the statistics fall within the predetermined ranges (yes), at 1104, the AI ​​processor may continue to perform the training process to generate multiple statistics for the model until the model converges through a predetermined number of training steps. However, if the statistics do not fall within the predetermined ranges (no), at 1108, the AI ​​processor may be configured to perform the training process with adjusted training parameters to keep the statistics within the predetermined ranges. Then, returning to 1104, the AI ​​processor may again perform the training process to generate multiple statistics in the model.

[0046]

[0071] The techniques described above can be implemented within a wide variety of computer systems configured to process artificial neural networks. FIG. 12 illustrates a simplified block diagram of an example computer system 1200 that can be used to implement the techniques described in the above disclosure. In some embodiments, the computer system 1200 can be used to implement, for example, the control processor 102. As shown in FIG. 12, the computer system 1200 includes one or more processors 1202 that communicate with several peripheral devices via a bus subsystem 1204. These peripheral devices may include a storage subsystem 1206 (including, for example, a memory subsystem 1208 and a file storage subsystem 1210) and a network interface subsystem 1216. Some computer systems may further include a user interface input device(s) 1212 and / or a user interface output device(s) 1214.

[0047]

[0072] Bus subsystem 1204 may provide a mechanism for allowing various components and subsystems of computer system 1200 to communicate with each other as desired. Although bus subsystem 1204 is shown generally as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses.

[0048]

[0073] Network interface subsystem 1216 may serve as an interface for communicating data between computer system 1200 and other computer systems or networks. Embodiments of network interface subsystem 1216 may include, for example, Ethernet, Wi-Fi, and / or cellular adapters, modems (telephone, satellite, cable, ISDN, etc.), digital subscriber line (DSL) units, and / or the like.

[0049]

[0074] Storage subsystem 1206 includes memory subsystem 1208 and / or file / disk storage subsystem 1210. Subsystems 1208 and 1210, as well as other memories described herein, are examples of non-transitory computer-readable storage media capable of storing executable program code and / or data that provide functionality for embodiments of the present disclosure.

[0050]

[0075] The memory subsystem 1208 includes several memories, including a main random access memory (RAM) 1218 for storing instructions and data during program execution, and a read-only memory (ROM) 1220 in which fixed instructions are stored. The file storage subsystem 1210 can provide persistent (e.g., non-volatile) storage for program and data files and can include magnetic or solid-state hard disk drives, optical drives with associated removable media (e.g., CD-ROM, DVD, Blu-Ray, etc.), removable flash memory-based drives or cards, and / or other types of storage media known in the art.

[0051]

[0076] It should be understood that computer system 1200 is exemplary and that many other configurations having more or fewer components than system 1200 are possible.

[0052]

[0077] FIG. 13 illustrates an artificial neural network processing system according to some embodiments. In various embodiments, a neural network according to the present disclosure (e.g., neural network 600) may be implemented and trained within a hardware environment including one or more neural network processors (e.g., AI processors). A neural network processor may refer to, for example, various graphics processing units (GPUs) (e.g., GPUs for processing neural networks manufactured by Nvidia Corp.®), field programmable gate arrays (FPGAs) (e.g., FPGAs for processing neural networks manufactured by Xilinx®), or various application-specific integrated circuits (ASICs) or neural network processors including hardware architectures optimized for neural network computation. In this example environment, one or more servers 1302, which may include the architecture shown in FIG. 12 above, may be coupled to multiple controllers 1310(1)-1310(M) over a communications network 1301 (e.g., a switch, router, etc.). The controllers 1310(1)-1310(M) may also include the architecture shown in FIG. 12 above. Each controller 1310(1)-1310(M) may be coupled to one or more neural network (NN) processors, such as processing units 1311(1)-1311(N) and 1312(1)-1312(N). The NN processing units 1311(1)-1311(N) and 1312(1)-1312(N) may include various configurations of functional processing blocks and memory optimized for neural network processing, such as training or inference. The NN processors are optimized for neural network computation. The server 1302 may configure the controller 1310 with NN models and input data to the models, which may be loaded and executed in parallel by, for example, the NN processing units 1311(1)-1311(N) and 1312(1)-1312(N). The models may include, for example, layers and associated weights, as described above. The NN processing units may load the models and apply the inputs to produce output results.The NN processing unit may also implement, for example, the training algorithms described herein.

[0053]

[0078] 14 illustrates an example of a graph 1400 illustrating optimization of validation loss (e.g., prediction accuracy) versus computational resources and time for training (e.g., petaflops / s-days * cost per unit of computation), according to one embodiment. In this example, a first dashed line 1402 (shown as the right-most downward dashed line) is a plot of the computationally optimal Pareto frontier for models of various sizes associated with higher precision or bit width (e.g., FP32). The models shown intersecting line 1402 are, for example, 10-bit models as shown by curve 1402a. 11 From the parameters (e.g., weights), the 10 5 The models have a variety of model sizes, ranging from 1402a to 1402b. Models with many parameters (e.g., 1402a) achieve lower validation loss (e.g., higher prediction accuracy) than models with fewer parameters (e.g., 1402b). However, models with many parameters (e.g., 1402a) require more computational resources and time to process than models with fewer parameters (e.g., 1402b).

[0054]

[0079] The computationally optimal Pareto frontier can be shifted to a second dashed line 1403 (shown as the intermediate downward dashed line) by reducing precision (e.g., 7 bits). Similarly, reducing precision to 3 bits can result in another Pareto frontier, shown by dotted line 1404. In each case, precision increases as model size scales, but so does computation (and cost). However, this example shows that overlapping Pareto curves create a new, more optimal curve, shown at 1405. In Pareto curve 1405, reducing precision as model size increases can result in fewer computational resources for a given precision. Thus, some embodiments of the present disclosure can reduce precision in at least a portion of the model being trained as model size increases, e.g., to reduce the number of training steps (and computational cycles) to achieve a particular validation loss.

[0055] Characterization and Mapping Examples

[0080] As discussed herein, in various embodiments, the mapping implemented by a mapping system, such as mapping system 132 of Figure 1, can efficiently characterize the relationship between statistics and training parameters in order to adjust the training parameters to optimize the training of a neural network. The following are examples of network statistics that can be mapped to training parameters to improve training:

[0056]

[0081] In a first example, when training a larger Transformer model (e.g., 20 million parameters) with six layers, gradient noise (e.g., statistics) can be monitored and controlled by selectively adjusting training parameters. For a model, the following trends in gradient noise can be characterized: gradient noise can stabilize within a desired threshold after 1-10% of the total number of training steps for convergence; reducing precision (e.g., reducing the number of bits in quantization) can increase gradient noise; reducing the learning rate can increase gradient noise; increasing the learning rate can decrease gradient noise; reducing batch size can increase gradient noise; and increasing batch size can decrease gradient noise. Thus, gradient noise can be controlled by modifying one or more of the above training parameters (e.g., precision, learning rate, and / or batch size). A target gradient noise for a given range can be calculated, for example, by Bcrit(L) = Bcrit(L). * can be determined using known closed-form estimates such as L / αB, where B * is about 2.10 8 token, αB is approximately 0.21, where B represents the batch size, L represents the cross-entropy loss, and α represents the power exponent for scaling. Additionally or alternatively, the target gradient noise can be determined by monitoring training with a known configuration over a period of time.

[0057]

[0082] With reference to FIG. 15A, when training an example Transformer model at a high precision or bit width (e.g., 16 bits), the gradient noise is approximately 10 in each of the layers as shown. 4.5 15D, which may provide a graph 1502 showing the convergence of a high accuracy model at a given validation loss (e.g., prediction accuracy) over several training steps.

[0058]

[0083] Referring to Figure 15B, when training an example model at a low precision or bit width (e.g., 12 bits) and without adjusting training parameters (e.g., without optimization), gradient noise can increase significantly, especially in the first two layers, as shown. Referring again to Figure 15D, this can result in graph 1504 showing the divergence (e.g., loss of prediction accuracy) of a low-precision model without optimization over several training steps.

[0059]

[0084] However, in this example, we can optimize the low-accuracy model by adjusting one or more training parameters to control gradient noise. For example, we can scale the learning rate by a power series fit to the rate between the first and last layers in the network: f(x)=ax k We can use a layer-by-layer learning rate scale by fitting the equation to x=1 in the first layer and x=6 in the last layer. The learning rate is one training parameter that can be used to control gradient noise, and by scaling each layer relative to the first layer, we can control gradient noise for low precision models (e.g., 12-bit).

[0060]

[0085] Referring again to Figure 15D, this may yield a graph 1506 showing the convergence (e.g., increasing prediction accuracy) of the low-precision model with optimization over several training steps. Thus, adjusting the learning rate and / or weight decay within the layers of the low-precision model to control the gradients within the layers may result in successful convergence of the model. These relationships may be characterized for the model and mapped accordingly.

[0061]

[0086] In a second example, when training a Transformer model with two layers, the standard deviation of activations and the standard deviation of gradients can be monitored and controlled by adjusting the learning rate and / or weight decay (e.g., training parameters). With reference to Figure 16A, graph 1602 shows a model associated with a high precision or bit-width that results in convergence (e.g., a stable decrease in validation loss) over several training steps. However, graph 1604 shows a model associated with a low precision or bit-width (e.g., without optimization) that results in divergence (e.g., an increase in validation loss) over several training steps.

[0062]

[0087] 16B, the standard deviation of layer activations (e.g., within the attention layer of a Transformer model) can be compared between a high-precision model, graph 1612, and a low-precision model (e.g., without optimization), graph 1614. Specifically, the low-precision model incurs a significant increase in node activations when compared to the high-precision model. Thus, to control and optimize the low-precision model, one can characterize the relationship between the standard deviation of layer activations and training parameters such as learning rate and / or weight decay.

[0063]

[0088] 16C, the standard deviation of layer gradients can be compared between a high-precision model, graph 1622, and a low-precision model (e.g., without optimization), graph 1624. Specifically, the low-precision model incurs significantly smaller gradients in the early portions of training when compared to the high-precision model. Therefore, to control and optimize the low-precision model, the relationship between the standard deviation of layer gradients and training parameters such as learning rate and / or weight decay can be characterized.

[0064]

[0089] 17A-B show the relationship between precision and percentage of zeros. With respect to FIG. 17A, graph 1702 shows loss curves for different precisions (or bit widths) over several training steps. Plot 1704 shows models associated with lower precision or bit widths (e.g., without optimization) that result in divergence (e.g., increased validation loss) over several training steps. Additional graphs of models associated with various levels of precision are shown between graphs 1702 and 1704.

[0065]

[0090] 17B, ​​the percentage of zero values ​​in a layer's activations (e.g., the SelfAttention layer) can be compared between a low-precision model (e.g., without optimization), graph 1714, and models with increasing levels of precision. Specifically, the low-precision model contains a larger percentage of zeros compared to the models with increasing levels of precision.

[0066]

[0091] Finally, with further reference to Figure 17C, the percentage of zero values ​​in a layer's activations (e.g., a multilayer perceptron) can be compared between a low accuracy model (e.g., without optimization) and a model with increasing accuracy levels, graph 1724. Specifically, the low accuracy model contains a larger percentage of zeros compared to the model with increasing accuracy levels, in this case 35% of the activation values ​​are zero in this layer at approximately 60,000 iterations.

[0067]

[0092] Therefore, to control and optimize training, we can characterize these relationships between the percentage of zero-valued activations of certain layers and training parameters.

[0068] Further exemplary embodiments

[0093] In various embodiments, the present disclosure includes systems, methods, and apparatus for optimizing artificial neural networks. The techniques described herein can be implemented by a non-transitory machine-readable medium storing a program executable by a computer system, the program including a set of instructions for performing the techniques described herein. In some embodiments, a system includes one or more control processors and a non-transitory machine-readable medium storing instructions that, when executed by at least one of the one or more control processors, cause the at least one control processor to perform the techniques. In some embodiments, the non-transitory machine-readable medium can be, for example, a memory that can be coupled to, for example, the one or more control processors or one or more artificial intelligence processors.

[0069]

[0094] The following techniques can be implemented alone or in various combinations, and can also be implemented in conjunction with other techniques described herein.

[0070]

[0095] For example, in one embodiment, the disclosure includes one or more artificial intelligence (AI) processors, one or more control processors, and a non-transitory computer-readable storage medium storing program code executable by the one or more control processors, the program code causing the one or more control processors to receive an artificial neural network model, configure the model based on a plurality of training parameters to perform a training process on the one or more AI processors, monitor a plurality of statistics resulting from the performance of the training process, and adjust one or more of the plurality of training parameters based on one or more statistics of the plurality of statistics to keep at least one of the plurality of statistics within a predetermined range.

[0071]

[0096] In one embodiment, the adjusting step includes adjusting the accuracy associated with the target stratum of the model to keep at least one of a plurality of statistics associated with the target stratum within a predetermined range.

[0072]

[0097] In one embodiment, the system further reduces the accuracy of at least a portion of the model as the size of the model increases.

[0073]

[0098] In one embodiment, the monitoring step includes monitoring one or more statistics associated with the target layer of the model.

[0074]

[0099] In one embodiment, the one or more statistics include a measure of the gradient noise of the neural network.

[0075]

[0100] In one embodiment, the adjusting step includes adjusting one or more training parameters associated with the target layer of the model.

[0076]

[0101] In one embodiment, the target layer is a hidden layer, in which the one or more training parameters include parameters that configure the accuracy of the target layer.

[0077]

[0102] In one embodiment, the target layer is a hidden layer, in which the one or more training parameters include a parameter that configures the sparsity of the target layer.

[0078]

[0103] In one embodiment, the target layer is a hidden layer, in which the one or more training parameters include a parameter configuring the number of nodes associated with the target layer.

[0079]

[0104] In one embodiment, the target layer is a hidden layer, in which the one or more training parameters include a parameter that configures the number of layers associated with the model.

[0080]

[0105] In one embodiment, the plurality of training parameters includes parameters constituting an accuracy associated with the model, wherein the model is comprised of a first accuracy associated with a first layer of the model and a second accuracy associated with a second layer of the model, the first accuracy being greater than the second accuracy, the first layer being closer to the input of the model, and the second layer being closer to the output of the model.

[0081]

[0106] In one embodiment, the plurality of training parameters includes a parameter configuring sparsity, in one embodiment, the model is configured with a first sparsity associated with a first layer of the model and a second sparsity associated with a second layer of the model, the first sparsity being different from the second sparsity, and the first layer being a layer closer to the input of the model and the second layer being a layer closer to the output of the model.

[0082]

[0107] In one embodiment, the model is a second model having a second size, and the training process is a second training process. In this embodiment, prior to the receiving step, the program code causes the one or more control processors to: receive the first model to perform the first training process; configure the first model based on the plurality of training parameters, where the first model is a smaller version of the second model, the first model has the size of the first model, and the size of the first model is significantly smaller than the size of the second model; monitor a plurality of statistics resulting from performing the first training process; and generate one or more mappings between the plurality of statistics and the plurality of training parameters.

[0083]

[0108] In one embodiment, to perform the second training process, the program code causes one or more control processors to load one or more mappings for adjusting one or more training parameters based on one or more statistics.

[0084]

[0109] In one embodiment, program code causes one or more control processors to load one or more mappings that include trained models that form relationships between statistics and parameters.

[0085]

[0110] In one embodiment, the plurality of training parameters includes one or more of a parameter configuring the size of the model, a parameter configuring the batch size, a parameter configuring the learning rate, a parameter configuring the precision, and a parameter configuring sparsity.

[0086]

[0111] In one embodiment, the plurality of statistics includes one or more of a measure of the gradient of the neural network, a measure of the gradient noise of the neural network, a measure of the activation of the nodes of the neural network, a measure of the weights of the neural network, and a measure of the updates of the weights of the neural network.

[0087]

[0112] In one embodiment, the one or more measurements include one or more of a mean measurement, a standard deviation measurement, and a percentage of zero values ​​measurement.

[0088]

[0113] In one embodiment, the training process is partitioned to run on multiple AI processors, and in this embodiment, measurements of multiple statistics are received from multiple AI processors.

[0089]

[0114] In one embodiment, the present disclosure includes one or more AI processors configured to perform a training process for an artificial neural network model, the training process having an associated set of training parameters, and performing the training process yields a plurality of statistics; and one or more control processors coupled to the one or more AI processors, the one or more control processors configured to receive the plurality of statistics and adjust one or more of the training parameters accordingly during performance of the training process to keep at least one of the plurality of statistics within a predetermined range.

[0090]

[0115] The above description illustrates various embodiments of the present disclosure, along with examples of how aspects of particular embodiments may be implemented. The above examples should not be considered the only embodiments, but are presented to illustrate the flexibility and advantages of particular embodiments as defined by the appended claims. Based on the above disclosure and the appended claims, other configurations, embodiments, implementations, and equivalents may be employed without departing from the scope of the present disclosure as defined by the appended claims.

Claims

1. one or more artificial intelligence (AI) processors; one or more control processors; a computer-readable storage medium storing program code executable by the one or more control processors; A system comprising: The program code receiving a first neural network model having a first model size; constructing the first neural network model based on training parameters and running a first training process on the one or more AI processors; monitoring a plurality of statistics of the first neural network model resulting from execution of the first training process by the one or more AI processors; generating one or more mappings characterizing relationships between the plurality of statistics and the training parameters; receiving a second neural network model having a second model size greater than the first model size; loading the one or more mappings to configure the second neural network model and running a second training process on the one or more AI processors with the training parameters adjusted according to the one or more mappings; causing the one or more control processors to system.

2. One or more artificial intelligence (AI) processors; one or more control processors; a computer-readable storage medium storing program code executable by the one or more control processors; A system comprising: The program code receiving an artificial neural network model and configuring the artificial neural network model based on a plurality of training parameters to perform a training process on the one or more AI processors; monitoring a plurality of statistics resulting from execution of said training process; adjusting one or more training parameters of the plurality of training parameters based on one or more statistics of the plurality of statistics to maintain at least one of the plurality of statistics within a predetermined range; causing the one or more control processors to perform 10. A system according to claim 9, wherein the plurality of training parameters includes parameters constituting an accuracy associated with the artificial neural network model, the artificial neural network model being configured with a first accuracy associated with a first layer of the artificial neural network model and a second accuracy associated with a second layer of the artificial neural network model, the first accuracy being greater than the second accuracy, the first layer being a layer closer to an input of the artificial neural network model, and the second layer being a layer closer to an output of the artificial neural network model.

3. One or more artificial intelligence (AI) processors; one or more control processors; a computer-readable storage medium storing program code executable by the one or more control processors; A system comprising: The program code receiving an artificial neural network model, configuring the artificial neural network model based on a plurality of training parameters, and executing a training process on the one or more AI processors; monitoring a plurality of statistics resulting from execution of said training process; adjusting one or more training parameters of the plurality of training parameters based on one or more statistics of the plurality of statistics to maintain at least one of the plurality of statistics within a predetermined range; causing the one or more control processors to perform 10. A system according to claim 9, wherein the plurality of training parameters includes a parameter configuring sparsity, the artificial neural network model being configured with a first sparsity associated with a first layer of the artificial neural network model and a second sparsity associated with a second layer of the artificial neural network model, the first sparsity being different from the second sparsity, the first layer being a layer closer to the input of the artificial neural network model and the second layer being a layer closer to the output of the artificial neural network model.

4. One or more artificial intelligence (AI) processors; one or more control processors; a computer-readable storage medium storing program code executable by the one or more control processors; A system comprising: The program code receiving an artificial neural network model, configuring the artificial neural network model based on a plurality of training parameters, and executing a training process on the one or more AI processors; monitoring a plurality of statistics resulting from execution of said training process; adjusting one or more training parameters of the plurality of training parameters based on one or more statistics of the plurality of statistics to maintain at least one of the plurality of statistics within a predetermined range; causing the one or more control processors to perform the artificial neural network model comprises a Transformer model in a natural language processing application; adjusting the one or more training parameters includes adjusting the one or more training parameters so that layers closer to the output of the artificial neural network model have higher gradient noise than layers closer to the input of the artificial neural network model.

5. 5. The system of claim 2, wherein the adjusting step comprises adjusting a precision associated with the target layer of the artificial neural network model to maintain at least one of the plurality of statistics associated with the target layer within the predetermined range.

6. The system of claim 2, further comprising reducing the accuracy of at least a portion of the artificial neural network model as the size of the artificial neural network model increases.

7. 5. The system of claim 2, wherein the monitoring step comprises monitoring one or more statistics associated with a target layer of the artificial neural network model.

8. The system of claim 7 , wherein the one or more statistics include a measure of neural network gradient noise.

9. 5. The system of claim 2, wherein the adjusting step comprises adjusting one or more training parameters associated with a target layer of the artificial neural network model.

10. The system of claim 9 , wherein the target layer is a hidden layer, and the one or more training parameters include parameters that configure the accuracy of the target layer.

11. 10. The system of claim 9, wherein the target layer is a hidden layer and the one or more training parameters include a parameter that configures sparsity of the target layer.

12. 10. The system of claim 9, wherein the target layer is a hidden layer, and the one or more training parameters include a parameter configuring a number of nodes associated with the target layer.

13. 10. The system of claim 9, wherein the target layer is a hidden layer and the one or more training parameters include a parameter configuring a learning rate for one or more nodes in the target layer.

14. 10. The system of claim 9, wherein the target layer is a hidden layer and the one or more training parameters include a parameter configuring the number of layers associated with the artificial neural network model.

15. the plurality of training parameters include one or more of a parameter configuring a model size, a parameter configuring a batch size, a parameter configuring a learning rate, a parameter configuring precision, and a parameter configuring sparsity; 5. The system of claim 2, wherein the plurality of statistics comprises one or more of a measure of neural network gradients, a measure of neural network gradient noise, a measure of neural network node activations, a measure of neural network weights, and a measure of neural network weight updates.

16. The system of claim 15 , wherein the one or more measurements include one or more of a mean value measurement, a standard deviation measurement, and a percentage of zero values ​​measurement.

17. A method for generating a neural network model, comprising: receiving a first neural network model having a first model size; and constructing the first neural network model based on training parameters and running a first training process on one or more AI processors; monitoring a plurality of statistics of the first neural network model resulting from execution of the first training process by the one or more AI processors; generating one or more mappings characterizing relationships between the plurality of statistics and the training parameters; receiving a second neural network model having a second model size greater than the first model size; loading the one or more mappings to configure the second neural network model and running a second training process on the one or more AI processors with the training parameters adjusted according to the one or more mappings; A method comprising:

18. A method for training a neural network comprising: receiving an artificial neural network model; configuring the artificial neural network model based on a plurality of training parameters; and executing a training process on one or more AI processors. monitoring a plurality of statistics resulting from execution of said training process; adjusting one or more training parameters of the plurality of training parameters based on one or more statistics of the plurality of statistics to maintain at least one of the plurality of statistics within a predetermined range; Including, 1. A method according to claim 1, wherein the plurality of training parameters includes parameters constituting an accuracy associated with the artificial neural network model, the artificial neural network model being configured with a first accuracy associated with a first layer of the artificial neural network model and a second accuracy associated with a second layer of the artificial neural network model, the first accuracy being higher than the second accuracy, the first layer being a layer closer to the input of the artificial neural network model, and the second layer being a layer closer to the output of the artificial neural network model.

19. A method for training a neural network comprising: receiving an artificial neural network model; configuring the artificial neural network model based on a plurality of training parameters; and executing a training process on one or more AI processors. monitoring a plurality of statistics resulting from execution of said training process; adjusting one or more training parameters of the plurality of training parameters based on one or more statistics of the plurality of statistics to maintain at least one of the plurality of statistics within a predetermined range; Including, 1. A method according to claim 1, wherein the plurality of training parameters includes a parameter constituting sparsity, the artificial neural network model being constructed with a first sparsity associated with a first layer of the artificial neural network model and a second sparsity associated with a second layer of the artificial neural network model, the first sparsity being different from the second sparsity, the first layer being a layer closer to the input of the artificial neural network model and the second layer being a layer closer to the output of the artificial neural network model.

20. A method for implementing a training process on one or more AI processors, comprising: receiving an artificial neural network model; configuring the artificial neural network model based on a plurality of training parameters; and monitoring a plurality of statistics resulting from execution of said training process; adjusting one or more training parameters of the plurality of training parameters based on one or more statistics of the plurality of statistics to maintain at least one of the plurality of statistics within a predetermined range; Including, the artificial neural network model comprises a Transformer model in a natural language processing application; 10. The method of claim 9, wherein adjusting the one or more training parameters comprises adjusting the one or more training parameters so that layers closer to an output of the artificial neural network model have higher gradient noise than layers closer to an input of the artificial neural network model.

Images