Sparsity masking method for neural network training
By generating a balanced sparse mask in deep neural networks and applying it during training, the problem of increased computing and storage requirements during DNN training is solved, and efficient sparse training is achieved, reducing training costs and improving performance.
Patent Information
- Application Number
- CN202280093114.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-29
- Filing Date
- 2022-12-26
- Publication Date
- 2024-11-08
AI Technical Summary
Deep neural networks (DNNs) have increased dramatically during training, resulting in slow and expensive training processes. It is difficult for existing sparse training techniques to effectively reduce end-to-end costs, especially when weight matrix transposition during backpropagation. The sparse mode fails, resulting in performance degradation.
A balanced sparse training method is proposed to ensure transposeability of sparse patterns and flexibly adjust sparseness during inference and backpropagation by generating balanced sparse masks to apply to the weight matrix during forward and backpropagation. level to achieve computation and storage optimization.
It effectively reduces training costs, improves training performance, achieves accuracy and performance close to intensive training, while simplifying hardware acceleration and reducing the use of computing and storage resources.
Smart Images

Figure CN118922841A_ABST
Abstract
Description
Background Art
[0001] Deep neural networks (DNNs) can be used in machine learning to build artificial intelligence models. Deep learning workloads include input data, weight matrices learned during supervised training, and activation matrices calculated based on the input data and weight matrices. As computing resources expand, larger data sets can be processed, which requires the DNN to be scaled up accordingly. Sparsity can be used as a tool to reduce the amount of computation and / or memory consumed for operations required during training of the DNN. Summary of the invention
[0002] This summary is provided to introduce some concepts in a simplified form, which are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in any part of this disclosure.
[0003] A method for training a neural network is proposed. For a weight matrix having an integer dimension M1 on a first dimension and an integer dimension M2 on a second dimension, a first balanced sparsity mask is generated, the first balanced sparsity mask being an N1 / M1 mask on the first dimension. The first balanced sparsity mask is applied to the weight matrix during inference. A second balanced sparsity mask is generated for the transpose of the weight matrix. The second balanced sparsity mask is an N2 / M2 mask on the second dimension. The second balanced sparsity mask is applied to the transpose of the weight matrix during back-propagation. BRIEF DESCRIPTION OF THE DRAWINGS
[0004] Figure 1 An example system for training a neural network is schematically shown.
[0005] Figure 2 An example of intensive training of a neural network is schematically shown.
[0006] Figure 3 An example of sparse training of a neural network is schematically shown.
[0007] Figure 4 A simple matrix sparsification is schematically shown.
[0008] Figure 5 Unstructured and balanced sparsity masks are schematically shown.
[0009] Figure 6 Sparsity masks with different degrees of sparsity are schematically shown.
[0010] Figure 7The transposed sparsity mask is schematically shown.
[0011] Figure 8 is a flowchart of an example method for sparse training of a neural network.
[0012] Fig. 9 A method for generating a dual N / M mask is schematically shown.
[0013] Fig.10 is a flowchart of an example method for sparse training of a neural network.
[0014] Fig.11 A method for applying a balanced N / M mask to weight matrix inference for inference is schematically illustrated.
[0015] Fig.12 A method for applying a balanced N / M mask to a transposed weight matrix for back-propagation is schematically shown.
[0016] Fig.13 is a flowchart of an example method for sparse ladder training for a neural network.
[0017] Fig.14 is a timing diagram for sparse step training of neural networks.
[0018] Fig.15 An example computing system is schematically depicted. DETAILED DESCRIPTION
[0019] As the number of parameters of deep neural networks (DNNs) increases dramatically, the computation and memory requirements for training these networks also increase. The training process becomes slow and computationally expensive. Sparseening over-parameterized DNNs is a common technique to reduce computation and memory usage during the inference phase. However, sparse training remains challenging.
[0020] By removing 50%, 75%, 87.5%, or more of each tensor in some, most, or all layers, the total amount of memory accesses and computation can be reduced accordingly. Unstructured sparsity techniques can allow for a high degree of accuracy, but are challenging to accelerate using hardware and are essentially limited to systems that operate with sparsity hard-coded. Other systems will simply assume dense training.
[0021] Therefore, in order to effectively reduce the end-to-end cost of training, sparsity should follow a certain pattern. For example, properly deploying balanced fine-grained sparsity can provide accuracy and performance that is asymptotically close to the accuracy of unstructured sparsity. In balanced fine-grained sparsity, a single MxM weight or other parameter block is constrained so that at most N of the M parameters in each row and / or column are non-zero. All other parameters are masked to 0. By constraining the sparsity pattern in this way, hardware can be more efficiently implemented to benefit from computational sparsity even for systems that do not inherently use sparsity.
[0022] One challenge is that during back-propagation, the weight matrix must be transposed. When the weight matrix is transposed, the row constraints of N / M no longer hold, and the resulting sparsity pattern is irregular. Creating a transposed mask that satisfies the usual N / M constraints is a computationally very difficult problem. Performing these calculations for every round of training can therefore significantly degrade performance without improving the performance of the resulting trained network.
[0023] This specific embodiment describes a system and method for balanced sparse training, and in particular, describes a method for accelerating the training phase by utilizing sparsity. During training, balanced sparsity can be applied to various parts of the network, including being applied to weights, activations, gradients, and / or error tensors. In a first set of examples, a mask is generated that can be used during both forward and reverse passes. In other words, the mask is valid for both the original tensor matrix and the transposed tensor matrix. In a second set of examples, the mask used for the forward pass is discarded, and a new mask is dynamically generated for the reverse pass. Additional adjustments to the balance speed and accuracy can be achieved by choosing to use only sparse weights during inference or during inference and reverse support, and using sparse or fully dense error and activation tensors.
[0024] In each example, training can be performed entirely in one sparsity setting, or in a collection of progressively less sparsity stages. In some examples, the network can be trained directly for sparse inference, and / or fine-tuned to a fully dense network at the end of sparse training. The training cost incurred for each approach is significantly lower than full dense training, and flexible constraints allow approximate dense training.
[0025] Figure 1 An example system 100 is shown for training a neural network 102. In this example, training data 104 is used to train parameters of the neural network 102, such as weights and / or gradients of the neural network 102. The training data 104 may be processed through multiple iterations to obtain a final set of trained model parameters.
[0026] The neural network 102 includes an input layer 110, one or more hidden layers 112, and an output layer 114. Each layer includes a plurality of nodes 120. A training supervisor 122 may provide training data 104 to the input layer 110 of the neural network 102. In some examples, the training data 104 may be divided into mini-batches and / or shards for distribution to subsets of inputs. The training supervisor 122 may include one or more network-accessible computing devices programmed to provide services responsible for managing resources for training jobs. The training supervisor 122 may further provide information and instructions related to the training process to each node 120.
[0027] In this example, nodes 120 of the model receive input values at input layer 110 during forward processing or inference and produce output results (125) at output layer 114. During training, data flows in the opposite direction during backpropagation (127), where the error between the network's results and the expected results is determined at the output and the weights are updated layer by layer, flowing from output layer 114 to input layer 110.
[0028] Each node 120 may include one or more agents 130 configured to supervise one or more workers 132. Typically, each node 120 includes multiple workers 132, and the agent 130 can monitor multiple workers. Each node may also include multiple agents 130. The node 120 may be implemented using a central processing unit (CPU), a GPU, a combination of a CPU and a GPU, or a combination of any CPU, GPU, ASIC, and / or other computer programmable hardware. The agents 130 and workers 132 within a common node 120 may share certain resources, such as one or more local networks, storage subsystems, local services, and the like.
[0029] Each agent 130 may include an agent processing unit 134, a training process 136, and an agent memory 138. Each worker 132 may include a worker processing unit 142 and a worker memory 144. Typically, the agent processing unit 134 is described as being implemented using a CPU, while the worker processing unit 142 is implemented using a GPU. However, other configurations are also possible. For example, some or all aspects may be additionally or alternatively implemented in a cloud computing environment. The cloud computing environment may include a model for implementing on-demand network access to a shared pool of configurable computing resources. This shared pool of configurable computing resources can be quickly provisioned via virtualization and then expanded accordingly. The cloud computing model may consist of various features, such as, for example, on-demand self-service, extensive network access, resource pooling, rapid elasticity, measured services, and the like.
[0030] A deep learning model (or "network") comprises a graph of parameterizable layers (or "operators") that together implement complex nonlinear functions. The network can be trained via a training data set that includes pairs of input examples (x) and outputs (y). The desired output is a learned function parameterized by weights (w) such that given an input (x), the prediction f(x; w) approaches (y).
[0031] Applying a function f(x; w) is performed by transforming the input (x) layer by layer to generate the output - a process known as inference. In a training setting, this is known as the forward pass. Provisioning a network to solve a specific task involves two stages - designing the network structure and training the network's weights. Once designed, the network structure generally does not change during the training process.
[0032] The training iteration begins with a forward pass, which is similar to inference, but in which the input to each layer is stored. The quality of the result f(x; w) of the forward pass is evaluated using the loss function to evaluate, to estimate the accuracy of the prediction. The following backward pass propagates the loss (e.g., error) from the last layer in the opposite direction. At each parameter layer (e.g., learnable layer), the backward pass uses the adjoint of the forward operation to compute the gradient g and uses the gradient g to reduce The learning rule is used to update the parameters or weights. This process is repeated iteratively for many examples until the function f(x; w) provides the desired accuracy.
[0033] As an example, Figure 2 A multi-layer neural network 200 is schematically shown, which includes an input layer (x0) 202, two hidden layers (x1) 204 and (x2) 206, and an output layer (x3) 208. In this example, the input layer 202 includes 5 neurons (210, 211, 212, 213, 214), the first hidden layer 204 includes 3 neurons (220, 221, 222), the second hidden layer 206 includes 4 neurons (230, 231, 232, 233), and the output layer 208 includes 3 neurons (241, 242, 243).
[0034] The neural network 200 includes an activation function, such as a rectified linear unit (not shown). The neural network 200 can be parameterized by weight matrices w1 250, w2 251, and w3 252 and a bias vector (not shown). Each weight matrix includes a weight for each connection between two adjacent layers. The forward pass can include a series of matrix-vector products f(x0; w), where x0 is an input vector or feature vector.
[0035] The size of deep neural networks, such as network 200, is rapidly outstripping the ability of hardware to quickly store and train deep neural networks. Sparsity can be applied to reduce the number of network parameters before, during, and after training by pruning edges from the underlying topology. Figure 3 A sparsified version 300 of network 200 is shown, which includes a hidden layer input layer (x0') 302, a hidden layer (x1') 304, and a hidden layer (x2') 306, and an output layer (x3') 308. In this example, the third input feature 212 and all its adjacent weights are removed from the input layer (x0') 302 (dashed lines). Additionally, hidden neurons 222 and 232 and their weights are removed from hidden layers (x1') 304 and hidden layers (x2') 306, respectively. Various other weights have been removed from the sparsified version 300, resulting in weight matrices (w1') 350, (w2') 351, and (w3') 352. Removing neurons or input features in this manner corresponds to removing rows or columns in the layer weight matrix. Removing individual weights corresponds to removing individual elements of the weight matrix. Sparsity can be derived or naturally generated and can be applied to other tensors and matrices, such as matrices for activations, errors, biases, etc.
[0036] For activations, turning off activation for a node essentially generates zero output. Sparsity as applied to activations works the same way, e.g., activations with higher magnitude have higher value to the network and are retained. In some examples, activations naturally approach sparsity, so true sparsity can be added with a modest effect.
[0037] Sparsifying the weight matrix or other matrices or tensors effectively reduces the complexity of the matrix multiplication events that utilize the matrix. The speed of matrix multiplication is directly related to the sparsity of the matrix. In order to obtain a certain level of efficiency, and therefore increase the processing speed, sparsity can be distributed between the two inputs of matmul. Applying 75% of the sparsity to the first matrix, and applying 0% of the sparsity to the second matrix, the process is accelerated by about 4 times. Another method to achieve a 4-fold speed increase is by applying 50% of the sparsity to the first matrix and 50% of the sparsity to the second matrix. Balance can therefore be achieved by distributing sparsity between weights and activations, between errors and activations, or to any two input matrices in the matmul operation. The use of regularization and lifting techniques can be used to distribute information across different blocks during training.
[0038] For example, in Figure 4In [the figure], a heatmap 410 of an 8x8 weight matrix to be sparsified is shown. Lighter shaded blocks represent higher values. A simple high-pass filter can be applied to obtain the highest values to form a sparsified matrix 420. However, using a simple filter like this leaves unbalanced rows and columns. Thus, while effectively reducing the complexity of any subsequent matrix multiplication, a more thoughtful sparsification method can further simplify the matrix, thereby allowing for more targeted matrix compression.
[0039] The compressed matrix and / or highly quantized values can be stored in memory like this and then inflated when retrieved and decompressed, e.g., for injection into a digital signal processor for computation. For example, activation sparsity can be used for activation storage and to improve memory storage and bandwidth during training. In the presence of activation and / or gradient sparsity, a learning rate higher than that usually used can potentially be used to improve model convergence.
[0040] For unstructured sparsity, the mask has few constraints and can essentially be configured in any random pattern. In Figure 5 [the figure], mask 510 is an example of unstructured sparsity. Each black square masks the underlying value to 0. Each white square allows the underlying value to be non-zero. The numbers on the grid axes are the counts for that row or column - e.g., how many non-zero values exist in that dimension. For example, the top row of mask 510 has one white square (non-zero value), and the second column from the left of mask 510 has two white squares (non-zero values). This convention is used throughout this disclosure.
[0041] Unstructured sparsity is typically applied after the network is trained, but in some cases can also be applied during training. Unstructured sparsity is the least constrained form of sparsity, but its inherent randomness makes it difficult to accelerate at the hardware level.
[0042] The most common constraint on balanced sparsity is the N / M constraint. Where, for a column or row with M values, only N (N < M) values can be non-zero. For example, mask 520 is an example of balanced sparsity with the value N = 1. Each row of mask 520 has one white square (non-zero value). The columns of mask 520 range from 0 to 2 non-zero values.
[0043] Balanced sparsity is thus more constrained than unstructured sparsity, but is easier to accelerate with hardware because the hardware will expect the results for each constrained row or column. The known constraints can be pre-loaded into the hardware. The best configuration for applying balanced sparsity can be based on the complexity of the AI application and the specifics of the underlying hardware. Balanced sparsity, in or by itself, does not limit the small-world property of the weights after convergence.
[0044] In this article, most examples can be considered balanced random fine-grained sparsity. In this context, fine-grained means that only a portion of the tensor is sparsified, while balanced means that all blocks (e.g., rows, columns) have the same sparsity level, but within each block, the pattern is random. This is in contrast to unstructured sparsity, in which the size of each sparsity block is equal to the size of the tensor. As the block size increases, the fidelity also increases because different configurations can be represented more flexibly. At least from a representation perspective, applying balanced sparsity with a sufficiently large block size effectively approximates unstructured sparsity. However, as the block size increases beyond a threshold, there are diminishing returns. Therefore, balanced random fine-grained sparsity represents an opportunity to strike a good balance between hardware implementation cost and accuracy.
[0045] Furthermore, balanced sparsity can be extended to different sparsity levels. As an example, Figure 6 Balanced sparsity masks of size M=8x8 are shown. Mask 600 has N / 1 along the rows, resulting in a mask with 87.5% sparsity. Mask 610 has N / 2 along the rows, resulting in a mask with 75% sparsity. Mask 620 has N / 3 along the rows, resulting in a mask with 50% sparsity. Mask 630 has N / 4 along the rows, resulting in a mask with 50% sparsity. Balanced sparsity can be applied to weights, activations, errors, and gradients, and can also have a scalable impact on training by selecting the tensors to sparsify.
[0046] However, since the weights must be transposed during backpropagation, the sparsity mask must take this transposition into account. However, even for simple 2D matrices such as Figure 5 and Figure 6 Finding transposable two-dimensional constraints is also challenging, as is the case with matrices such as those depicted in . Figure 7 Shown is a sparsity mask 710 of N / M(1 / 4) and its associated transposed mask 720. Even in this modest example, the row constraints for the transposed mask 720 are no longer as balanced as the rows in mask 710.
[0047] One way to quickly solve for unbalanced sparsity for transposed matrices is to apply the dual N / M constraint. Figure 8 In the example shown in , two N / M masks are derived - one is the target matrix, and one is the transpose of the target matrix, resulting in one mask with N / M sparsity in the row dimension and another mask with N / M sparsity in the column dimension. The two derived masks are then combined to generate a mask with less than or equal to N / M non-zero parameters in the row and column dimensions, making the mask transposable and enabling sparsity to be applied to the weight matrices in each layer during inference and back-propagation.
[0048] Figure 8 An example method 800 for training a neural network is shown. Method 800 may be performed by one or more computing systems, such as system 100 and / or neural network 200. More specifically, method 800 may be performed by a training supervisor, such as training supervisor 122, while communicating with a plurality of nodes that may be configured to train a machine learning model. A technical effect of implementing such a method is an improvement in the use of computing resources.
[0049] At 810, method 300 includes, for a weight matrix having an integer dimension J on a first dimension and an integer dimension K on a second dimension, reshaping the weight matrix into one or more square weight matrices having an integer dimension M on the first dimension and an integer dimension M on the second dimension. If J is already equal to K, the weight matrix may not need to be reshaped to perform element-wise multiplication. In some examples, in order to maintain all values within the weight matrix, two or more square matrices may be generated with dimensions MxM, M'xM', etc.
[0050] Each generated MxM weight matrix can then be processed separately. For each weight matrix, at 820, method 800 includes generating a first balanced sparsity mask, which is an N / M mask on a first dimension. At 830, method 800 includes generating a second balanced sparsity mask, which is an N / M mask in a second dimension for transposing the weight matrix. In some examples, the first balanced sparsity mask and the second balanced sparsity mask are generated based on at least a top-K function.
[0051] For example, in Fig. 9 In FIG. 9 , a heat map 900 of the original weight matrix to be sparse is shown. Lighter shaded blocks represent higher values. For the first sparsity matrix 910, the highest values are taken to form the target matrix. In other words, the filter aperture of the mask corresponds to the absolute magnitude of the values in the original block. Then, the N / M mask is derived from the transpose 920 of the first sparsity matrix 910.
[0052] At 840, method 800 includes combining the first balanced sparsity mask and the second balanced sparsity mask to generate a third sparsity mask. For example, combining the first sparsity mask and the second sparsity mask to generate the third sparsity mask may include combining the first sparsity mask and the second sparsity mask using an element-wise Boolean AND operation.
[0053] like Fig. 9As shown in , matrix 910 and matrix 920 are combined to generate matrix 930. For example, the matrices can be combined using a Boolean AND function. Matrix 930 can therefore be transposed in any way and has at most N / M constraints in any transposition. Since the mask is only transposed and not changed, it can be cached separately during the forward pass and reused or applied to the weight matrix to create the final sparse weight matrix, which can also be cached. During backpropagation, the cached mask / weights can be used directly without recalculating a new mask.
[0054] Other techniques for finding transposed N / M masks are inherently slow using linear solvers or greedy algorithms, which become exponentially slower as unstructured sparsity is approached through looser constraints and larger block sizes. In these examples, the method can utilize basic top-K sorting, which is suitable for medium-sized matrices. The trade-off for the speed gained in determining such a transposed mask is that sometimes the mask ends up being underfilled. Such a mask may have less non-zero values than the maximum allowed number of non-zero values. If the original mask and the transposed mask are different, values may be sacrificed and weights that may be affected may be lost.
[0055] Therefore, in some examples, method 800 may also include setting the desired output sparsity for the third sparsity mask, and generating a first balanced sparsity mask and a second balanced sparsity mask whose sparsity is greater than the desired output sparsity. For example, the desired output sparsity may be set to 50%. The first balanced sparsity mask and the second balanced sparsity mask may then be oversupplied at a sparsity of 70% or 75%. Additional top-K functions may be executed to obtain oversupplied masks. Therefore, the operator can strike a balance between the cost of any additional hardware complexity and the expected improvement in accuracy. For example, particularly compared to methods that require calculating multiple order gradients or may take multiple iterations to derive the most effective mask, where such filtering is applied at each iteration, the simplicity of calculating the mask offsets any loss in accuracy.
[0056] At 850, method 800 includes applying a third sparsity mask to the weight matrix during reasoning. At 850, method 800 includes applying the transposition of the third sparsity mask to the transposition of the weight matrix during back propagation. In addition, method 800 may include applying N / M sparsity to one or more tensors in the activation tensor and the error tensor during back propagation. This ensures that back propagation is completely sparse. Depending on system parameters, modality, and data set properties, sparsity can be applied to either matrix. Although the weight matrix is used for forward and backward propagation, and therefore must be transposed, activation does not need to be transposed, and error information is specific to reverse transmission.
[0057] Sparse training can also be used in an unstructured manner, improving training fidelity compared to structured sparse training. Memory usage and computational resources required during training can be reduced. However, in order to do this, the associated hardware must be very specialized, making it tricky to support non-sparse training on the same hardware.
[0058] N / M sparsity is easier to speed up than unstructured sparsity in terms of memory access and computation. However, generating a transposable mask can apply additional constraints on the weight, error, and / or activation matrices. This can further degrade training performance.
[0059] Fig.10 An example method 1000 for training a neural network is shown that enables structured sparse training to match the performance of N / M sparsity if the balancing constraint of the transposition is not considered. This method may still be applicable to non-sparse specific accelerator designs. This method may be called dynamic masking. Dynamic masking removes the transposition constraint on the weight mask, making structured sparse training more competitive with unstructured sparse training. Generating masks for dynamic sparsity may also be a thousand times faster or more than iteratively creating a perfectly optimized transposable mask using linear programming. Compared to existing sparse training methodologies (such as transposable masks), this improves the fidelity of training while still computing sparsity during backpropagation and without the added computational expense of generating complex masks for each pass or set of passes.
[0060] Method 1000 may be performed by one or more computing systems, such as system 100 and / or neural network 200. More specifically, method 1000 may be performed by a training supervisor, such as training supervisor 122, while communicating with a plurality of nodes that may be configured to train a machine learning model. The technical effect of implementing this method is an improvement in the use of computing resources.
[0061] At 1010, method 1000 includes, for a weight matrix having an integer dimension M1 in a first dimension and an integer dimension M2 in a second dimension, generating a first balanced sparsity mask, the first balanced sparsity mask being an N1 / M1 mask in the first dimension. Fig.11 , a heat map of the original weight matrix 1100 to be sparsified is shown. The N1 / M1 mask is applied to generate a sparse weight matrix 1120, where N1=1 and M1=4 along the rows.
[0062] Return to Fig.10At 1020, method 1000 includes applying a first balanced sparsity mask to the weight matrix during inference. The mask generated during the forward pass may just be a standard N / M mask, but it will not be cached. Instead, during backpropagation, a new N / M mask is created and applied using the transposed weight matrix, resulting in a row-by-row N / M sparse matrix. This matrix will have exactly N / M non-zero parameters.
[0063] At 1030, method 1000 includes generating a second balanced sparsity mask for the transpose of the weight matrix, the second balanced sparsity mask being an N2 / M2 mask in the second dimension. Fig.12 , a heat map is shown for weight matrix 1200, which is the transpose of weight matrix 1100. An N2 / M2 mask is applied to generate a sparse weight matrix 1220, where N=1 and M=4 along the rows.
[0064] In some examples, the weight matrix will be configured so that M1=M2. In some examples, the first balanced sparsity mask and the second balanced sparsity mask can use the same integer value for N1 and N2. In other words, the same principles and parameters for row selection apply to the second balanced sparsity mask and the first balanced sparsity mask. In some examples, the first balanced sparsity mask and the second balanced sparsity mask can be generated based on at least the top-K function. In some examples, the second balanced sparsity mask can be the transpose of the first balanced sparsity mask.
[0065] At 1040, method 1000 includes applying the second balanced sparsity mask to the transpose of the weight matrix during back propagation. In other words, the second balanced sparsity mask can be used to perform gradient calculations during back propagation. Therefore, there is no need to use a lossy approximation of the transposable sparsity.
[0066] Performing masking in this way does not necessarily represent a straight-through estimate of the gradients. For example, some influential weights may be passed through incorrectly, while these weights are not used during inference, or vice versa. However, any errors that occur should have a marginal impact on training accuracy or loss compared to using an unbalanced masking scheme that cannot be accelerated at the hardware level. Dynamic masking can significantly improve the fidelity of the sparse matrix to the original dense matrix, resulting in a marginal impact on training, rather than using a technically correct but unbalanced transposed matrix from the forward pass.
[0067] Since sparsity itself is an approximation of dense training, the additional approximation may not significantly affect training. Specifically, during the initial stages of training, the model is performing more exploratory operations, working at a more reasonable granularity. Therefore, the small amount of noise is drowned out by the quantization that has already occurred. Sparsity can be used on top of quantization to enhance the compression efficiency of the compression scheme. Therefore, sparsity can not only improve computational utilization, but also memory storage / bandwidth and communication.
[0068] At the end of training, adding noise or sparsity may cause the model to deviate from the local minimum in a promising subspace. Therefore, it becomes clearer how much noise and sparsity have been added. An option for maintaining speed and computational resources at the beginning of training is to perform sparse ladder training, where accuracy is less important, but improve accuracy in later training stages.
[0069] In any of the systems and methods for sparse training described herein, sparse ladder training can be used to speed up and reduce training costs without affecting the accuracy of the resulting training model. Sparse ladder training can gradually reduce, increase, or cycle sparsity during training according to appropriate training goals. For example, at the beginning of training, when the network is exploring, it can be trained with relatively high sparsity to achieve speed improvements without affecting the final performance level. If high sparsity affects performance, for example, training slows down, then sparsity can be reduced during training.
[0070] Compared to standard dense training, sparse ladder training can generate efficiency improvements by exploiting how quickly the loss decreases at the beginning of training. Because the network is randomly initialized, initial progress is very fast in the early stages of learning, regardless of the actual capacity of the network. As training continues, the network begins to fill its capacity, and learning progresses more slowly. During the initial fast learning phase, high sparsity levels can be used to speed up training without significantly affecting loss. When the network reaches its capacity, sparsity can be reduced, thereby increasing capacity and further improving model loss. This process can be repeated until the network performs as a fully dense network while using much less compute than fully dense training.
[0071] Fig.13 An example method 1300 for sparse ladder training is shown. The method 1300 may be performed by one or more computing systems, such as the system 100 and / or the neural network 200. More specifically, the method 1300 may be performed by a training supervisor, such as the training supervisor 122, while communicating with a plurality of nodes that may be configured to train a machine learning model. A technical effect of implementing this method is an improvement in the use of computing resources.
[0072] At 1310, method 1300 includes performing training over a plurality of consecutive stages, each stage comprising a plurality of rounds, and wherein a first stage of training is initially performed using an initial sparsity level. As an example, an early training stage can be accelerated by using a high sparsity level (e.g., 87.5%, 75%) for weights and / or for one or both of an activation tensor and an error tensor. The sparsity level and tensors selected to be sparsified can be viewed as tunable hyperparameters that can be adjusted to achieve a balance between trading training speed and training accuracy.
[0073] As an example, Fig.14 A hypothetical timing diagram 1400 for performing sparse step training is shown. The timing diagram 1400 includes a graph 1402 indicating the sparsity level over multiple consecutive training rounds and a graph 1404 indicating the gradient noise (normalized to arbitrary units) over multiple consecutive training rounds. An increase in gradient noise is associated with a decrease in training performance. At time t0, training is performed in a first stage 1405 at a sparsity level of 87.5%.
[0074] Return to Fig.13 At 1320, method 1300 includes determining a measure of training performance after each round. The sparsity level can be switched based on a threshold change in gradient noise, which statistic may be significantly and proportionally affected by sparsity. The threshold change may be a deviation from a known baseline for a particular model type. This increase may correspond to training that performs worse than a representative baseline and would therefore benefit from switching to a lower sparsity level. Therefore, a maximum allowed threshold for gradient noise may be set at one or more layers of the network. The degree of sparsity may therefore be dynamically switched accordingly in order to maintain the desired training performance compared to a dense baseline. Additionally or alternatively, as training progresses, the rate of change of the loss will decrease (e.g., flatten out), possibly reaching within a threshold of 0, and may be used as an indicator for switching the sparsity level.
[0075] In some examples, a step-wise schedule can be selected before initiating training, where it switches at the sparsity level executed within an auto-scheduling interval (e.g., at the end of a predetermined number of iterations). Such a schedule can be determined based on a power-law analysis for sparse networks of different sizes, such that different sparsity levels are implemented based on the trade-off between network performance and capacity and thus the extent to which sparse training can recover the performance of full dense training. By noting where sparse training changes a power law compared to a dense network, such as a power law for training efficiency, a step-wise schedule can be constructed to optimize both sparsity and the total amount of computation required.
[0076] Return to Fig.13At 1330, method 1300 includes, in response to a measure of training performance falling below a first threshold, advancing training to a second stage by adjusting the initial sparsity level to a reduced sparsity level. As an example, during training, the sparsity level may be reduced from 75% to 50% (e.g., N / M increases), and training restarted. As sparsity decreases, the network capacity is higher, and the loss may begin to decrease again.
[0077] For example, Fig.14 As shown in , the gradient noise increases to the first threshold 1407 at time t1. Therefore, the training enters the second stage 1410, and the sparsity level is reduced from 87.5% to 75%. A new gradient noise threshold 1412 is established for measuring the training performance in the second stage 1410.
[0078] Return to Fig.13 At 1340, method 1300 includes, in response to the metric that the training performance drops below a second threshold, advancing the training to a third stage by adjusting the reduced sparsity level to a further reduced sparsity level.
[0079] For example, Fig.14 As shown in , the gradient noise increases to the second threshold 1412 at time t2. Therefore, the training enters the third stage 1415, and the sparsity level is reduced from 75% to 50%. A new gradient noise threshold 1417 is established to measure the training performance in the third stage 1415. Any number of stages and thresholds can be established. In some examples, such as Fig.14 As shown in , a static threshold can be applied instead of an adjustable threshold. As shown, the gradient noise increases to the third threshold 1417 at time t3. Therefore, the training enters the fourth stage 1420, and the sparsity level is reduced from 50% to 25%. A new gradient noise threshold 1422 is established for measuring the training performance in the fourth stage 1420.
[0080] Return to Fig.13 At 1350, method 1300 optionally includes, in response to a metric of training performance falling below a final threshold, advancing training to a final stage by adjusting the sparsity level to a fully dense level at a further reduced level. If fully dense performance is desired, a small fully dense fine-tuning may be used to restore the full accuracy of the network. In some examples, such as for very large networks or networks to which lower sparsity levels have already been applied, a dense fine-tuning stage is not required and sparse performance is reasonably close to dense performance. Fig.14As shown in , the gradient noise increases to a fourth threshold 1422 at time t4. Therefore, the training enters the fifth stage 1425, and the sparsity level is reduced from 25% to 0%. By adjusting the sparsity in this way, the accuracy of the training is gradually restored at each stage using a less sparse matrix, and the early calculation level is reduced by using a higher sparsity level.
[0081] The step-wise approach can be applied to one or more of the passes, for example, only to the backpropagation pass. For example, the backpropagation pass may be more susceptible to sparsity than the inference pass. In one example, the network may initially be trained at 75% sparsity in both the forward and backward passes. Subsequently, the sparsity for the backward pass may be gradually reduced independently of the forward pass. In this way, the training performance is improved in terms of loss, but does not need to be slower than the forward pass at lower sparsity levels. This can allow a sparse network to be trained in a single iteration without a separate intensive fine-tuning step to recover the accuracy lost due to sparsity.
[0082] Weights are typically easy to sparsify, and, for example, activations may be less sensitive to sparsity than error terms and backpropagation. In addition, different parameters and directions may be sparsified to different degrees, depending on the platform on which the model is intended to be used. In scenarios where the model is trained to perform reasoning in a sparse setting, it may be desirable to perform sparse step training only on the back pass (e.g., error terms, gradient terms, activations), while keeping the forward pass at a fixed degree of sparsification. Thus, when training is complete, the model can be implemented in a sparse configuration. If the model is being trained to perform dense reasoning, sparse step training can be applied to both the back pass and the forward pass to improve accuracy at all levels.
[0083] Specifically, by implementing sparse ladder training, time savings are achieved in terms of computation and memory. For example, at 75% sparsity, depending on the hardware design, a training step may complete four times faster than an equivalent step in fully dense training. By using ladders with lower sparsity, a baseline performance level can be achieved in a reduced total time period.
[0084] The implementation for sparse training described herein has the technical effect of reducing computational costs in the training and development of machine learning models, thereby reducing the cost of acquiring such models for consumers. Sparse neural networks trained with sparsity can be used directly for sparse inference without any additional fine-tuning stages, which is often necessary to meet latency and cost targets.
[0085] An end user may be enabled to control the degree of sparsification during inference. Depending on the intended hardware platform, network, or system used to implement the trained model, during the forward pass of training, the training constraints may be adjusted accordingly. Activation terms and error terms may be sparsified without regard to transposition. Thus, the sparsity level of these terms may be set based on a speed versus fidelity tradeoff compared to intensive training. The technical effect of the method is a more customizable software package that may be obtained without an expensive, time-consuming, intensive training procedure.
[0086] In some embodiments, the methods and processes described herein may be related to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as a computer application or service, an application programming interface (API), a library, and / or other computer program products.
[0087] Fig.15 Schematically illustrated is a non-limiting embodiment of a computing system 1500 that can implement one or more of the above methods and processes. Computing system 1500 is shown in simplified form. Computing system 1500 can take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices. System 100, system 200, and system 300 can be examples of computing system 1500.
[0088] The computing system 1500 includes a logic machine 1510 and a storage machine 1520. The computing system 1500 may optionally include a display subsystem 1530, an input subsystem 1540, a communication subsystem 1550, and / or Fig.15 Other components not shown.
[0089] Logic machine 1510 includes one or more physical devices configured to execute instructions. For example, the logic machine can be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical structures. Such instructions can be implemented to perform tasks, implement data types, transform the state of one or more components, implement technical effects, or otherwise achieve a desired result.
[0090] The logic machine may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machine may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processor of the logic machine may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel and / or distributed processing. The individual components of the logic machine may optionally be distributed between two or more separate devices, which may be remotely located and / or configured for coordinated processing. Various aspects of the logic machine may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.
[0091] In addition to including one or more GPUs 1554, the logic subsystem may also include one or more CPUs 1552, and the one or more CPUs 1552 may be configured to send executable instructions and / or data to the one or more GPUs 1554. In response to processing the instructions and / or data by the one or more GPUs 1554, the CPUs 1552 may receive result data from the one or more GPUs 1554. In this way, the logic subsystem may perform a large number of calculations in parallel via the GPUs. Specifically, the logic subsystem may efficiently perform Figure 8 Method 800, Fig.10 Method 1000 and Fig.13 Method 1300.
[0092] The present disclosure refers to the GPU as a computing device that is well suited for the distributed learning process because the GPU is configured to execute a large number of multiple replicated instances of the same program (e.g., GPU kernel) in parallel, where each instance of the program receives and processes different input data. However, it is to be understood that other aspects of the logic subsystem can be configured to provide the same or similar advantages. Therefore, it is to be understood that any discussion of the GPU also applies to other suitable computing components, and the present disclosure is not in any way limited to executing method 800, method 1000, method 1300, or any other aspect of training a machine learning model on a GPU to exclude other suitable computing devices.
[0093] The storage machine 1520 includes one or more physical devices configured to store instructions executable by the logic machine to implement the methods and processes described herein. When such methods and processes are implemented, the state of the storage machine 1520 can be transformed to, for example, store different data.
[0094] The storage machine 1520 may include removable and / or built-in devices. The storage machine 1520 may include optical storage (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor storage (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic storage (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), etc. The storage machine 1520 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices.
[0095] It will be understood that storage 1520 includes one or more physical devices. However, certain aspects of the instructions described herein may alternatively be propagated by a communication medium (eg, electromagnetic signals, optical signals, etc.) that is not held by a physical device for a limited duration.
[0096] Aspects of the logic machine 1510 and the storage machine 1520 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include field programmable gate arrays (FPGAs), programming and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).
[0097] The terms "module", "program", and "engine" may be used to describe aspects of the computing system 1500 implemented to perform specific functions. In some cases, a module, program, or engine may be instantiated via the logic machine 1510 executing instructions stored by the storage machine 1520. It will be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module", "program", and "engine" may cover individual executable files, data files, libraries, drivers, scripts, database records, etc. or groups of executable files, data files, libraries, drivers, scripts, database records, etc.
[0098] It will be understood that a "service" as used herein is an application program that is executable across multiple user sessions. A service may be available for one or more system components, programs, and / or other services. In some implementations, a service may run on one or more server computing devices.
[0099] When included, display subsystem 1530 can be used to present a visual representation of the data stored by storage machine 1520. This visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data stored by the storage machine, and thus transform the state of the storage machine, the state of display subsystem 1530 can also be transformed to visually represent the changes in the underlying data. Display subsystem 1530 may include one or more display devices utilizing almost any type of technology. Such display devices can be combined with logic machine 1510 and / or storage machine 1520 in a shared housing, or such display devices can be peripheral display devices.
[0100] When included, the input subsystem 1540 may include or interface with one or more user input devices (e.g., a keyboard, mouse, touch screen, or game controller). In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the transduction and / or processing of input actions may be handled on-board or off-board. Example NUI components may include microphones for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity.
[0101] When included, the communication subsystem 1550 can be configured to communicatively couple the computing system 1500 with one or more other computing devices. The communication subsystem 1550 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem can be configured to communicate via a wireless telephone network, or a wired or wireless local area network or wide area network. In some embodiments, the communication subsystem can allow the computing system 1500 to send messages to other devices and / or receive messages from other devices via a network such as the Internet.
[0102] In one example, a method for training an operating neural network includes: generating a first balanced sparsity mask for a weight matrix having an integer dimension M1 on a first dimension and an integer dimension M2 on a second dimension, the first balanced sparsity mask being an N1 / M1 mask on the first dimension; applying the first balanced sparsity mask to the weight matrix during inference; generating a second balanced sparsity mask for the transpose of the weight matrix, the second balanced sparsity mask being an N2 / M2 mask on the second dimension; and applying the second balanced sparsity mask to the transpose of the weight matrix during back propagation. In this example or any other example, N1 and N2 are additionally or alternatively the same integer value. In any of the foregoing examples or any other example, the second balanced sparsity mask is additionally or alternatively the transpose of the first balanced sparsity mask. In any of the foregoing examples or any other example, the first balanced sparsity mask and the second balanced sparsity mask are additionally or alternatively generated based on at least a top-K function. In any or any other example of the foregoing examples, training is additionally or alternatively performed in a plurality of sequential stages, each stage comprising a plurality of rounds, and wherein the first sequential training stage is additionally or alternatively performed using an initial sparsity level. In any or any other example of the foregoing examples, the method additionally or alternatively includes determining a measure of training performance after each round. In any or any other example of the foregoing examples, the method additionally or alternatively includes advancing the training to a second stage by adjusting the initial sparsity level to a reduced sparsity level in response to the measure of training performance decreasing below a first threshold. In any or any other example of the foregoing examples, the method additionally or alternatively includes advancing the training to a third stage by adjusting the reduced sparsity level to a further reduced sparsity level in response to the measure of training performance decreasing below a second threshold. The technical effect of implementing this method is an improvement in the use of computing resources.
[0103] In another example, a method for training a neural network includes: for a weight matrix having an integer dimension J on a first dimension and an integer dimension K on a second dimension, reshaping the weight matrix into one or more weight matrices having an integer dimension M on the first dimension and an integer dimension M on the second dimension; for each weight matrix: generating a first balanced sparsity mask, the first balanced sparsity mask being an N / M mask on the first dimension; for the transpose of the weight matrix, generating a second balanced sparsity mask, the second balanced sparsity mask being an N / M mask on the second dimension; combining the first balanced sparsity mask and the second balanced sparsity mask to generate a third sparsity mask; during inference, applying the third sparsity mask to the weight matrix; and during backpropagation, applying the transpose of the third sparsity mask to the transpose of the weight matrix. In this example or any other example, the first balanced sparsity mask and the second balanced sparsity mask are additionally or alternatively generated based on at least a top-K function. In any or any other example of the foregoing examples, the method additionally or alternatively includes setting the desired output sparsity for the third sparsity mask; and generating a first balanced sparsity mask and a second balanced sparsity mask when the sparsity is greater than the desired output sparsity. In any or any other example of the foregoing examples, combining the first sparsity mask and the second sparsity mask to generate the third sparsity mask additionally or alternatively includes combining the first sparsity mask and the second sparsity mask using an element-wise Boolean AND operation. In any or any other example of the foregoing examples, the method additionally or alternatively includes applying N / M sparsity to one or more of the activation tensor and the error tensor during backpropagation. In any or any other example of the foregoing examples, the training is additionally or alternatively performed in multiple sequential stages, each stage including multiple rounds, and wherein the first stage training is additionally or alternatively performed using an initial sparsity level. In any or any other example of the foregoing examples, the method additionally or alternatively includes determining a measure of training performance after each round. In any or any other of the foregoing examples, the method additionally or alternatively includes advancing the training to a second stage by adjusting the initial sparsity level to a reduced sparsity level in response to the measure of training performance decreasing below a first threshold. In any or any other of the foregoing examples, the method additionally or alternatively includes advancing the training to a third stage by adjusting the reduced sparsity level to a further reduced sparsity level in response to the measure of training performance decreasing below a second threshold. A technical effect of implementing this method is to reduce computational costs in training and developing machine learning models.
[0104] In yet another example, a computing system for operating a deep neural network includes one or more logic machines; and one or more storage machines, each storing instructions that, when executed by the one or more logic machines, cause the computing system to: generate a first balanced sparsity mask for a weight matrix having an integer dimension M on a first dimension and having an integer dimension M on a second dimension, the first balanced sparsity mask being an N / M mask on the first dimension; during a forward pass, apply the first balanced sparsity mask to the weight matrix; generate a second balanced sparsity mask for a transpose of the weight matrix, the second balanced sparsity mask being an N / M mask on the second dimension; and during a backward pass, apply the second balanced sparsity mask to the transpose of the weight matrix. In this example or any other example, the second balanced sparsity mask is additionally or alternatively a transpose of the first balanced sparsity mask. In any of the foregoing examples or any other examples, the storage machine additionally or alternatively stores instructions that, when executed by one or more logic machines, cause the computing system to: perform training in a plurality of sequential stages, each stage comprising a plurality of rounds, and wherein the first stage training is performed using an initial sparsity level; after each round, determine a measure of training performance; and in response to the measure of training performance decreasing below a first threshold, advance the training to a second stage by adjusting the initial sparsity level to a reduced sparsity level. The technical effect of implementing this method is an improvement in the use of computing resources.
[0105] It will be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples will not be considered in a limiting sense, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the various behaviors illustrated and / or described may be performed in the order illustrated and / or described, in other orders, in parallel, or omitted. Likewise, the order of the above process may be changed.
[0106] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts and / or properties disclosed herein, as well as any and all equivalents thereof.
Claims
1. A method for training a neural network, comprising: For a weight matrix having an integer dimension M1 in a first dimension and an integer dimension M2 in a second dimension, generating a first balanced sparsity mask, wherein the first balanced sparsity mask is an N1 / M1 mask in the first dimension; During inference, applying the first balanced sparsity mask to the weight matrix; For the transpose of the weight matrix, generating a second balanced sparsity mask, where the second balanced sparsity mask is an N2 / M2 mask on the second dimension; as well as During back-propagation, the second balanced sparsity mask is applied to the transpose of the weight matrix.
2. The method of claim 1, wherein N1 and N2 are the same integer value. The method of claim 1 , wherein the second balanced sparsity mask is the transpose of the first balanced sparsity mask. The method according to claim 1 , wherein the first balanced sparsity mask and the second balanced sparsity mask are generated based on at least a top-K function.
5. The method of claim 1, wherein the training is performed in a plurality of sequential stages, each stage comprising a plurality of rounds, and wherein a first sequential stage of training is performed using an initial sparsity level.
6. The method according to claim 5, further comprising: After each episode, a training performance metric is determined.
7. The method according to claim 6, further comprising: In response to the training performance metric decreasing below a first threshold, the training is advanced to a second stage by adjusting the initial sparsity level to a decreased sparsity level.
8. The method according to claim 7, further comprising: In response to the training performance metric decreasing below a second threshold, the training is advanced to a third stage by adjusting the decreased sparsity level to a further decreased sparsity level.
9. A computing system for training a deep neural network, comprising: one or more logical machines; as well as One or more storage machines, each storing instructions that, when executed by the one or more logic machines, cause the computing system to: For a weight matrix having an integer dimension J on a first dimension and an integer dimension K on a second dimension, reshape the weight matrix into one or more weight square matrices having an integer dimension M on the first dimension and an integer dimension M on the second dimension; For each weight matrix: generating a first balanced sparsity mask, wherein the first balanced sparsity mask is an N / M mask in the first dimension; For the transpose of the weight matrix, generating a second balanced sparsity mask, where the second balanced sparsity mask is an N / M mask in the second dimension; combining the first balanced sparsity mask and the second balanced sparsity mask to generate a third sparsity mask; During inference, applying the third sparsity mask to the weight matrix; as well as During back-propagation, the transpose of the third sparsity mask is applied to the transpose of the weight matrix. 10 . The computing system of claim 9 , wherein the first balanced sparsity mask and the second balanced sparsity mask are generated based on at least a top-K function.
11. The computing system of claim 9, wherein the storage machine further stores instructions that, when executed by the one or more logic machines, cause the computing system to: setting a desired output sparsity for the third sparsity mask; and When the sparsity is greater than the desired output sparsity, the first balanced sparsity mask and the second balanced sparsity mask are generated.
12. The computing system of claim 9, wherein combining the first sparsity mask and the second sparsity mask to generate the third sparsity mask comprises: The first sparsity mask and the second sparsity mask are combined using an element-wise Boolean AND operation.
13. The computing system of claim 9, wherein the storage machine further stores instructions that, when executed by the one or more logic machines, cause the computing system to: During the back-propagation, N / M sparsity is applied to one or more of the activation tensor and the error tensor.
14. The computing system of claim 9, wherein the training is performed in a plurality of sequential stages, each stage comprising a plurality of rounds, and wherein a first stage of training is performed using an initial sparsity level.
15. The computing system of claim 14, wherein the storage machine further stores instructions that, when executed by the one or more logic machines, cause the computing system to: After each epoch, determining a training performance metric; and In response to the training performance metric decreasing below a first threshold, the training is advanced to a second stage by adjusting the initial sparsity level to a decreased sparsity level.
